Trjtdtk

I am using Gensim Phrases to detct n-grams in my text. Thus, I am interested in knowing the mechanism that Phrases uses to detect these n-grams in the text. Can someone please explain the mechanism used in Phrases in simple terms?

As the gensim tool cites the very famous paper by Mikolov - "Distributed Representations of Words and Phrases..." using which it is implemented. In the paper if you look at the section "4 Learning Phrases" they give a nice explanation of how n-grams are calculated (Equation 6).

So, if want to count bigrams this formula is straight-forward; score(wi, wj) is the score between any two words occuring together. But when counting trigrams, 'wi' will be a bigram and 'wj' will be a word. And same follows for any number of grams.

Gensim detects a bigram if a scoring function for two words exceeds a threshold (which is a parameter for Phrases).

The default scoring function is what is in the answer by flyingDope, but multiplied by vocabulary size (use help(Phraser) or see the gensim's Github repository (gensim/models/phrases.py)):

def original_scorer(worda_count, wordb_count, bigram_count, len_vocab, min_count, corpus_word_count):
 #...
 """
 worda_count : int
 Number of occurrences for first word.
 wordb_count : int
 Number of occurrences for second word.
 bigram_count : int
 Number of co-occurrences for phrase "worda_wordb".
 len_vocab : int
 Size of vocabulary.
 min_count: int
 Minimum collocation count threshold.
 corpus_word_count : int
 Not used in this particular scoring technique.
 """
 #...

 return (bigram_count - min_count) / worda_count / wordb_count * len_vocab

Another implemented score function is npmi_scorer based on a paper by G. Bouma.

I think n-grams for n>2 are done by applying bigram detection n-1 times.

If min_count (i.e. $delta$) was zero and if instead len_vocab we multiplied by corpus_word_count, then the result of original_scorer would be essentially the ratio of the probability to see wordb following worda and the unconditional probability to see wordb at a random position, that is how many times the presence of worda increases the probability to see wordb in the next position.

I cannot understand why gensim chose to use len_vocab here, but perhaps they had some reason to. You can pass your own scoring function as well.