Does gensim.corpora.Dictionary have term frequency saved?

No, gensim.corpora.Dictionary does not save term frequency. You can see the source code here. The class only stores the following member variables:

    self.token2id = {}  # token -> tokenId
    self.id2token = {}  # reverse mapping for token2id; only formed on request, to save memory
    self.dfs = {}  # document frequencies: tokenId -> in how many documents this token appeared

    self.num_docs = 0  # number of documents processed
    self.num_pos = 0  # total number of corpus positions
    self.num_nnz = 0  # total number of non-zeroes in the BOW matrix

This means everything in the class defines frequency as document frequency, never term frequency, as the latter is never stored globally. This applies to filter_n_most_frequent(remove_n) as well as every other method.


I had the same simple question. It appears that the frequency of the word is hidden and not accessible in the object. Not sure why it makes testing and validation a pain. What I did was export the dictionary as text..

dictionary.save_as_text('c:\\research\\gensimDictionary.txt')

In that text file they have three columns.. For example here are the words "summit" "summon" and "sumo"

Key Word Frequency

10 summit 1227

3658 summon 118

8477 sumo 40

I found a solution the .cfs are the word frequencies.. see https://radimrehurek.com/gensim/corpora/dictionary.html#gensim.corpora.dictionary.Dictionary

print(str(dictionary[10]), str(dictionary.cfs[10])) 

summit 1227

simple