List the words in a vocabulary according to occurrence in a text corpus, with Scikit-Learn CountVectorizer
If cv
is your CountVectorizer
and X
is the vectorized corpus, then
zip(cv.get_feature_names(),
np.asarray(X.sum(axis=0)).ravel())
returns a list of (term, frequency)
pairs for each distinct term in the corpus that the CountVectorizer
extracted.
(The little asarray
+ ravel
dance is needed to work around some quirks in scipy.sparse
.)
There is no built-in. I have found a faster way to do it based on Ando Saabas's answer:
from sklearn.feature_extraction.text import CountVectorizer
texts = ["Hello world", "Python makes a better world"]
vec = CountVectorizer().fit(texts)
bag_of_words = vec.transform(texts)
sum_words = bag_of_words.sum(axis=0)
words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]
sorted(words_freq, key = lambda x: x[1], reverse=True)
output
[('world', 2), ('python', 1), ('hello', 1), ('better', 1), ('makes', 1)]