Trjtdtk

How to output the tokens produced using MWE Tokenizer?

NLTK's multi-word expression tokenizer (MWETokenizer) provides a method/function add_mwe() that allows the user to enter multiple word expressions prior to using the tokenizer on the text.

Currently, I have a file consisting of phrases / multi-word expression I want to use with the tokenizer. My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text.

So this leads me to ask if anyone knows how to output the token generated by add_mwe() so that I can verify that I am correctly passing the phrase to the function?

You can check the exact input and output parameters of the add_mwe method in NLTK's documentation for the class here.

This is the expected input:

>>> tokenizer.add_mwe(('in', 'spite', 'of'))

So, each phrase must simply be a tuple with the words in that phrase. If you provide that input, you should get the output you expect (in_spite_of). I've added a full snippet of working code below for convenience, there you can see how to use the class as intended.

Regarding the output of add_mwe, every time you call the method it adds a new word to the dictionary, and all the words are stored in the class's _mwes attribute. So, given mwe = MWETokenizer(), you can then inspect the contents of mwe (with e.g. print mwe._mwes) to see what the class actually stores.

As stated in the documentation, it is actually a Trie with all the terms, so it won't look exactly as the words you added (it is a more efficient representation thereof). The link I mentioned earlier has more details on that.

Hope this helps!

import nltk

from nltk import (
 sent_tokenize as splitter,
 wordpunct_tokenize as tokenizer
)

from nltk.tokenize.mwe import MWETokenizer

test = """Anyone know how to output the tokens produced using MWE Tokenizer?

For a clearer explanation of what I am asking for those who did not understand my original brief question.

The multi-word expression tokenizer (MWETokenizer) provides a method/function (add_mwe()) that allows the user to enter multiple word expressions prior to using the tokenizer on text. Currently I have a file consisting of phrases / multi-word expression I want to use with the tokenizer. My concern is that the manner in which I am presenting the phrases to the function correctly and so not resulting in the desired set of tokens to be used in tokenizing the incoming text. So this leads me to ask if anyone knows how to output the token generated by this method/function so that I can verify that I am correctly passing the phrase to the function (add_mwe()).?"""

mwe = MWETokenizer()

phrases = [
 ('multi', '-', 'word'),
 ('expression', 'tokenizer'),
 ('word', 'expressions'),
 ('multi', '-', 'word', 'expression')
]

for phrase in phrases:
 mwe.add_mwe(phrase)


for sent in splitter(test):
 tokens = tokenizer(sent)
 print ' '.join(tokens)
 print ' '.join(mwe.tokenize(tokens))
 print '---'



# Expected output:
#
# Anyone know how to output the tokens produced using MWE Tokenizer ?
# Anyone know how to output the tokens produced using MWE Tokenizer ?
# ---
# For a clearer explanation of what I am asking for those who did not understand my original brief question .
# For a clearer explanation of what I am asking for those who did not understand my original brief question .
# ---
# The multi - word expression tokenizer ( MWETokenizer ) provides a method / function ( add_mwe ()) that allows the user to enter multiple word expressions prior to using the tokenizer on text .
# The multi_-_word_expression tokenizer ( MWETokenizer ) provides a method / function ( add_mwe ()) that allows the user to enter multiple word_expressions prior to using the tokenizer on text .
# ---
# Currently I have a file consisting of phrases / multi - word expression I want to use with the tokenizer .
# Currently I have a file consisting of phrases / multi_-_word_expression I want to use with the tokenizer .
# ---
# ...