X-Y Fast Trie in real world applications
While templatetypedef seems to have given a detailed answer, most points in that answer are simply not true. Let me refute it one by one:
- X-fast tries internally require several linked structures...don't perform well in database environments...
X-fast tries are essentially binary search trees defined on metric space. There are techniques to organize a binary trees into reasonable pages. The double linked structure seems bad, but the pointer will not be invoked until a successor was found. That's only 1 disk seek.
- X-fast tries internally require the use of perfect hash tables...
That's not true. You can use whatever hashing techniques. To guarantee the constant time look up, you can use cuckoo hashing instead of perfect hashing.
- Because of the hash tables in X-fast tries...only amortized efficient rather than worst-case efficient...
That's not true either. If you use perfect-hashing or cuckoo hashing, constant look-up time will be guaranteed. The insert/delete time will be amortized, but that's not too bad for a search intensive system (e.g. Lucene). Most modern search engines are built on the idea of sacrificing write time to boost search.
- The use of hash tables in X-fast tries and Y-fast tries means that there is an element of randomness involved in the runtimes of the data structures...
That's depend on the goal of the system. The above statement is true only if you want to support reliable and efficient write. But for the majority of the search-oriented applications, your priority is to boost the search speed.
- Due to all the reasons outlined above, the constant factors buried in the runtimes of X-fast and Y-fast tries are quite large...
It is true that the constant factor is not trivial - but I would be surprised to see anything more than 4, and consider the loglogM time (e.g., 6 for a 64 bit universe), that's nothing.
So what's the real reason why there are so few industrial applications of Y-fast tries? The database industry is a large, lucrative business that tends to be slow in adopting new techniques. The R-tree did not see much adoption until the late 1990s. The object-oriented concepts were consistently rejected by the industry, despite its theoretical maturity. They DB people also kept rejecting anything other than B-trees until open source search engines such as Lucene beats RDBMS in almost all fronts of search-related business. Just last month a senior Oracle guy told me a trie/hash table based system can never be real time - until I show him how to do it with in-memory caching/merging.
There are a few reasons that X-fast or Y-fast tries might not be useful in practice. Here are a few:
X-fast tries internally require several linked structures, including a bitwise trie and a doubly-linked list of elements. These don't perform well in database environments where elements are stored on disks and following a pointer can require a disk seek. (For similar reasons, databases often use B-trees over binary search trees). Additionally, they require the use of balanced binary search trees augmented with information to perform a split or join, which adds in extra space and introduces even more pointers to follow.
X-fast tries internally require the use of hash tables with worst-case O(1) lookups. Hash tables with these requirements usually require a variety of hash functions to be applied to look up an element and (generally speaking) don't have the best locality compared to, say, a linear-probing hash table, so lookups are a bit slower.
Because of the hash tables in X-fast tries and the use of splitting and joining of BSTs in Y-fast tries, these two data structures are only amortized efficient rather than worst-case efficient. In some cases, this is unacceptable - it would be bad if, periodically, a database query ends up taking 100x or 1000x normal time even if on average everything works out quite well.
The use of hash tables in X-fast tries and Y-fast tries means that there is an element of randomness involved in the runtimes of the data structures. On expectation they're efficient, but it's possible that due to bad luck, the runtimes of the data structures might be quite high. Specifically, the cost of doing a rehash in an internal hash table or doing a split or join on a tree can be quite high. In a database implementation, reliability is important, so this randomness might hurt.
Due to all the reasons outlined above, the constant factors buried in the runtimes of X-fast and Y-fast tries are quite large. In the long run, they should end up being faster than other data structures, but "the long run" might require inputs that are vastly larger than the sorts of data sets that could feasibly fit into a database.
Hope this helps!