The principle of LM deep model The Next CEO of Stack Overflow2019 Community Moderator ElectionN-grams in NLP deep learningWhat principle is behind semantic segmenation with CNNs?How do we pass data to a RNN?what machine/deep learning/ nlp techniques are used to classify a given words as name, mobile number, address, email, state, county, city etcText Classification with deep learningEncoder-Decoder Sequence-to-Sequence Model for Translations in Both DirectionsWhy ELMo's word embedding can represent the word better than glove?Build an Autocomplete model for document titlesHow to train neural word embeddings?What is the difference between TextGAN and LM for text generation?
Indicator light circuit
Extending anchors in TikZ
Complex fractions
Is there a difference between "Fahrstuhl" and "Aufzug"
What do "high sea" and "carry" mean in this sentence?
What can we do to stop prior company from asking us questions?
How to prepend a string to only the lines of text which are numbers
Why do professional authors make "consistency" mistakes? And how to avoid them?
How can I get through very long and very dry, but also very useful technical documents when learning a new tool?
Grabbing quick drinks
Help understanding this unsettling image of Titan, Epimetheus, and Saturn's rings?
What happens if you roll doubles 3 times then land on "Go to jail?"
How easy is it to start Magic from scratch?
How to make a software documentation "officially" citable?
Why is there a PLL in CPU?
What happened in Rome, when the western empire "fell"?
Extracting names from filename in bash
Can a caster that cast Polymorph on themselves stop concentrating at any point even if their Int is low?
What does "Its cash flow is deeply negative" mean?
Can the Reverse Gravity spell affect the Meteor Swarm spell?
Robert Sheckley short story about vacation spots being overwhelmed
How do you know when two objects are so called entangled?
Anatomically Correct Strange Women In Ponds Distributing Swords
Clustering points and summing up attributes per cluster in QGIS
The principle of LM deep model
The Next CEO of Stack Overflow2019 Community Moderator ElectionN-grams in NLP deep learningWhat principle is behind semantic segmenation with CNNs?How do we pass data to a RNN?what machine/deep learning/ nlp techniques are used to classify a given words as name, mobile number, address, email, state, county, city etcText Classification with deep learningEncoder-Decoder Sequence-to-Sequence Model for Translations in Both DirectionsWhy ELMo's word embedding can represent the word better than glove?Build an Autocomplete model for document titlesHow to train neural word embeddings?What is the difference between TextGAN and LM for text generation?
$begingroup$
Language model(LM) is the task of predicting the next word.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Or both with-encoder and without-encoder can do the LM task?
neural-network deep-learning natural-language-process language-model transformer
asked Mar 22 at 9:35
不是phd的phd不是phd的phd
2049
$endgroup$
add a comment |
$begingroup$
Language model(LM) is the task of predicting the next word.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Or both with-encoder and without-encoder can do the LM task?
neural-network deep-learning natural-language-process language-model transformer
asked Mar 22 at 9:35
不是phd的phd不是phd的phd
2049
$endgroup$
add a comment |
$begingroup$
Language model(LM) is the task of predicting the next word.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Or both with-encoder and without-encoder can do the LM task?
neural-network deep-learning natural-language-process language-model transformer
asked Mar 22 at 9:35
不是phd的phd不是phd的phd
2049
$endgroup$
Language model(LM) is the task of predicting the next word.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Or both with-encoder and without-encoder can do the LM task?
neural-network deep-learning natural-language-process language-model transformer
neural-network deep-learning natural-language-process language-model transformer
asked Mar 22 at 9:35
不是phd的phd不是phd的phd
2049
asked Mar 22 at 9:35
不是phd的phd不是phd的phd
2049
asked Mar 22 at 9:35
不是phd的phd不是phd的phd
2049
asked Mar 22 at 9:35
不是phd的phd不是phd的phd
2049
asked Mar 22 at 9:35
不是phd的phd不是phd的phd
2049
2049
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
The goal of LM is to learn a probability distribution over sequences
of symbols pertaining to a language.
That is, to learn $P(w_1,...,w_N)$ (resource).
This modeling can be accomplished by
- Predicting the next word given the previous words: $P(w_i | w_1,...,w_i-1)$, or
- Predicting the neighbor words given the center word (Skip-gram): $P(w_i+k| w_i), k in -2, -1, 1, 2$, or
- Predicting the center word given the neighbor words (CBOW or Continuous Bag-of-Words): $P(w_i| w_i-2,w_i-1,w_i+1, w_i+2)$, or other designs.
Does the deep model need the encoder? From the ptb code of
tensor2tensor, I find the deep model do not contains the encoder.
Yes. Modern LM solutions (all deep ones) try to find an encoding (embedding) that helps them to predict the next, neighbor, or center words as close as possible. However, a word encoding can be used as a constant input to other models. The ptb.py code calls text_encoder.TokenTextEncoder to receive such word encodings.
Both with-encoder and without-encoder can do the LM task?
LM task can be tackled without encoders too. For example, we can use frequency tables of adjacent words to build a model (n-gram modeling); e.g. all pairs (We, ?) appeared 10K times, pair (We, can) appeared 100 times, so P(can | We) = 0.01. However, encoder is the core of modern LM solutions.
answered Mar 22 at 11:27
EsmailianEsmailian
2,048218
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
return StackExchange.using("mathjaxEditing", function ()
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix)
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
);
);
, "mathjax-editing");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47773%2fthe-principle-of-lm-deep-model%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The goal of LM is to learn a probability distribution over sequences
of symbols pertaining to a language.
That is, to learn $P(w_1,...,w_N)$ (resource).
This modeling can be accomplished by
- Predicting the next word given the previous words: $P(w_i | w_1,...,w_i-1)$, or
- Predicting the neighbor words given the center word (Skip-gram): $P(w_i+k| w_i), k in -2, -1, 1, 2$, or
- Predicting the center word given the neighbor words (CBOW or Continuous Bag-of-Words): $P(w_i| w_i-2,w_i-1,w_i+1, w_i+2)$, or other designs.
Does the deep model need the encoder? From the ptb code of
tensor2tensor, I find the deep model do not contains the encoder.
Yes. Modern LM solutions (all deep ones) try to find an encoding (embedding) that helps them to predict the next, neighbor, or center words as close as possible. However, a word encoding can be used as a constant input to other models. The ptb.py code calls text_encoder.TokenTextEncoder to receive such word encodings.
Both with-encoder and without-encoder can do the LM task?
LM task can be tackled without encoders too. For example, we can use frequency tables of adjacent words to build a model (n-gram modeling); e.g. all pairs (We, ?) appeared 10K times, pair (We, can) appeared 100 times, so P(can | We) = 0.01. However, encoder is the core of modern LM solutions.
answered Mar 22 at 11:27
EsmailianEsmailian
2,048218
$endgroup$
add a comment |
$begingroup$
The goal of LM is to learn a probability distribution over sequences
of symbols pertaining to a language.
That is, to learn $P(w_1,...,w_N)$ (resource).
This modeling can be accomplished by
- Predicting the next word given the previous words: $P(w_i | w_1,...,w_i-1)$, or
- Predicting the neighbor words given the center word (Skip-gram): $P(w_i+k| w_i), k in -2, -1, 1, 2$, or
- Predicting the center word given the neighbor words (CBOW or Continuous Bag-of-Words): $P(w_i| w_i-2,w_i-1,w_i+1, w_i+2)$, or other designs.
Does the deep model need the encoder? From the ptb code of
tensor2tensor, I find the deep model do not contains the encoder.
Yes. Modern LM solutions (all deep ones) try to find an encoding (embedding) that helps them to predict the next, neighbor, or center words as close as possible. However, a word encoding can be used as a constant input to other models. The ptb.py code calls text_encoder.TokenTextEncoder to receive such word encodings.
Both with-encoder and without-encoder can do the LM task?
LM task can be tackled without encoders too. For example, we can use frequency tables of adjacent words to build a model (n-gram modeling); e.g. all pairs (We, ?) appeared 10K times, pair (We, can) appeared 100 times, so P(can | We) = 0.01. However, encoder is the core of modern LM solutions.
answered Mar 22 at 11:27
EsmailianEsmailian
2,048218
$endgroup$
add a comment |
$begingroup$
The goal of LM is to learn a probability distribution over sequences
of symbols pertaining to a language.
That is, to learn $P(w_1,...,w_N)$ (resource).
This modeling can be accomplished by
- Predicting the next word given the previous words: $P(w_i | w_1,...,w_i-1)$, or
- Predicting the neighbor words given the center word (Skip-gram): $P(w_i+k| w_i), k in -2, -1, 1, 2$, or
- Predicting the center word given the neighbor words (CBOW or Continuous Bag-of-Words): $P(w_i| w_i-2,w_i-1,w_i+1, w_i+2)$, or other designs.
Does the deep model need the encoder? From the ptb code of
tensor2tensor, I find the deep model do not contains the encoder.
Yes. Modern LM solutions (all deep ones) try to find an encoding (embedding) that helps them to predict the next, neighbor, or center words as close as possible. However, a word encoding can be used as a constant input to other models. The ptb.py code calls text_encoder.TokenTextEncoder to receive such word encodings.
Both with-encoder and without-encoder can do the LM task?
LM task can be tackled without encoders too. For example, we can use frequency tables of adjacent words to build a model (n-gram modeling); e.g. all pairs (We, ?) appeared 10K times, pair (We, can) appeared 100 times, so P(can | We) = 0.01. However, encoder is the core of modern LM solutions.
answered Mar 22 at 11:27
EsmailianEsmailian
2,048218
$endgroup$
The goal of LM is to learn a probability distribution over sequences
of symbols pertaining to a language.
That is, to learn $P(w_1,...,w_N)$ (resource).
This modeling can be accomplished by
- Predicting the next word given the previous words: $P(w_i | w_1,...,w_i-1)$, or
- Predicting the neighbor words given the center word (Skip-gram): $P(w_i+k| w_i), k in -2, -1, 1, 2$, or
- Predicting the center word given the neighbor words (CBOW or Continuous Bag-of-Words): $P(w_i| w_i-2,w_i-1,w_i+1, w_i+2)$, or other designs.
Does the deep model need the encoder? From the ptb code of
tensor2tensor, I find the deep model do not contains the encoder.
Yes. Modern LM solutions (all deep ones) try to find an encoding (embedding) that helps them to predict the next, neighbor, or center words as close as possible. However, a word encoding can be used as a constant input to other models. The ptb.py code calls text_encoder.TokenTextEncoder to receive such word encodings.
Both with-encoder and without-encoder can do the LM task?
LM task can be tackled without encoders too. For example, we can use frequency tables of adjacent words to build a model (n-gram modeling); e.g. all pairs (We, ?) appeared 10K times, pair (We, can) appeared 100 times, so P(can | We) = 0.01. However, encoder is the core of modern LM solutions.
answered Mar 22 at 11:27
EsmailianEsmailian
2,048218
edited Mar 22 at 14:40
answered Mar 22 at 11:27
EsmailianEsmailian
2,048218
answered Mar 22 at 11:27
EsmailianEsmailian
2,048218
answered Mar 22 at 11:27
EsmailianEsmailian
2,048218
2,048218
add a comment |
add a comment |
Thanks for contributing an answer to Data Science Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f47773%2fthe-principle-of-lm-deep-model%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
