Trjtdtk

Why are the trig functions versine, haversine, exsecant, etc, rarely used in modern mathematics?

Trademark violation for app?

Is this homebrew Lady of Pain warlock patron balanced?

How to write the following sign?

Circuit to "zoom in" on mV fluctuations of a DC signal?

Take 2! Is this homebrew Lady of Pain warlock patron balanced?

Dating a Former Employee

What is the appropriate index architecture when forced to implement IsDeleted (soft deletes)?

How would a mousetrap for use in space work?

Is CEO the "profession" with the most psychopaths?

How come Sam didn't become Lord of Horn Hill?

Can a new player join a group only when a new campaign starts?

Has negative voting ever been officially implemented in elections, or seriously proposed, or even studied?

Modified Intersection Puzzle

Did Deadpool rescue all of the X-Force?

Is there a kind of relay only consumes power when switching?

As a beginner, should I get a Squier Strat with a SSS config or a HSS?

Why do we bend a book to keep it straight?

Fundamental Solution of the Pell Equation

Project Euler #1 in C++

If windows 7 doesn't support WSL, then what does Linux subsystem option mean?

Would "destroying" Wurmcoil Engine prevent its tokens from being created?

What causes the direction of lightning flashes?

Why is Nikon 1.4g better when Nikon 1.8g is sharper?

Vanishing gradient problem for recent stochastic recurrent neural networks

0

$begingroup$

Recently, I've found some papers about generative recurrent models. All have attached sub-networks like prior/encoder/decoder/etc. to well-known LSTM cell for composing an aggregation of new-type RNN cell.

I am just curious about whether the gradient vanishing/exploding happens or not to those new RNN cell. Isn't there any problem about that kind of combination?

References:

It seems like they all have similar pattern as mentioned above.

A Recurrent Latent Variable Model for Sequential Data

Learning Stochastic Recurrent Networks

Z-Forcing: Training Stochastic Recurrent Networks

Pseudocode

The pseudocode for recurrent architecture is below:

def new_rnncell_call(x, htm1):
 #prior_net/posterior_net/decoder_net is single layer or mlp each
 q_prior = prior_net(htm1) # prior step
 q = posterior_net([htm1, x]) # inference step
 z = sample_from(q) # reparameterization trick
 target_dist = decoder_net(z) # generation step
 ht = innerLSTM([z, x], htm1) # recurrent step
 return [q_prior, q, target_dist], ht

What concerns me are those naked weights outside of well-known LSTM (or GRU etc.) cell during processing bptt without any gating logic for activations as the weights inside LSTM. For me, this looks not similar to stacked-rnn layers or additional dense layers just to outputs.

Doesn't that have any gradient vanishing/exploding problem?

python deep-learning gradient-descent recurrent-neural-net

share|improve this question

asked Apr 3 at 4:48

Sehee ParkSehee Park

11

$endgroup$

add a comment | 

0

$begingroup$

Recently, I've found some papers about generative recurrent models. All have attached sub-networks like prior/encoder/decoder/etc. to well-known LSTM cell for composing an aggregation of new-type RNN cell.

I am just curious about whether the gradient vanishing/exploding happens or not to those new RNN cell. Isn't there any problem about that kind of combination?

References:

It seems like they all have similar pattern as mentioned above.

A Recurrent Latent Variable Model for Sequential Data

Learning Stochastic Recurrent Networks

Z-Forcing: Training Stochastic Recurrent Networks

Pseudocode

The pseudocode for recurrent architecture is below:

def new_rnncell_call(x, htm1):
 #prior_net/posterior_net/decoder_net is single layer or mlp each
 q_prior = prior_net(htm1) # prior step
 q = posterior_net([htm1, x]) # inference step
 z = sample_from(q) # reparameterization trick
 target_dist = decoder_net(z) # generation step
 ht = innerLSTM([z, x], htm1) # recurrent step
 return [q_prior, q, target_dist], ht

What concerns me are those naked weights outside of well-known LSTM (or GRU etc.) cell during processing bptt without any gating logic for activations as the weights inside LSTM. For me, this looks not similar to stacked-rnn layers or additional dense layers just to outputs.

Doesn't that have any gradient vanishing/exploding problem?

python deep-learning gradient-descent recurrent-neural-net

share|improve this question

asked Apr 3 at 4:48

Sehee ParkSehee Park

11

$endgroup$

add a comment | 

$begingroup$

Recently, I've found some papers about generative recurrent models. All have attached sub-networks like prior/encoder/decoder/etc. to well-known LSTM cell for composing an aggregation of new-type RNN cell.

I am just curious about whether the gradient vanishing/exploding happens or not to those new RNN cell. Isn't there any problem about that kind of combination?

References:

It seems like they all have similar pattern as mentioned above.

A Recurrent Latent Variable Model for Sequential Data

Learning Stochastic Recurrent Networks

Z-Forcing: Training Stochastic Recurrent Networks

Pseudocode

The pseudocode for recurrent architecture is below:

def new_rnncell_call(x, htm1):
 #prior_net/posterior_net/decoder_net is single layer or mlp each
 q_prior = prior_net(htm1) # prior step
 q = posterior_net([htm1, x]) # inference step
 z = sample_from(q) # reparameterization trick
 target_dist = decoder_net(z) # generation step
 ht = innerLSTM([z, x], htm1) # recurrent step
 return [q_prior, q, target_dist], ht

What concerns me are those naked weights outside of well-known LSTM (or GRU etc.) cell during processing bptt without any gating logic for activations as the weights inside LSTM. For me, this looks not similar to stacked-rnn layers or additional dense layers just to outputs.

Doesn't that have any gradient vanishing/exploding problem?

python deep-learning gradient-descent recurrent-neural-net

share|improve this question

asked Apr 3 at 4:48

Sehee ParkSehee Park

11

$endgroup$

def new_rnncell_call(x, htm1):
 #prior_net/posterior_net/decoder_net is single layer or mlp each
 q_prior = prior_net(htm1) # prior step
 q = posterior_net([htm1, x]) # inference step
 z = sample_from(q) # reparameterization trick
 target_dist = decoder_net(z) # generation step
 ht = innerLSTM([z, x], htm1) # recurrent step
 return [q_prior, q, target_dist], ht

share|improve this question

asked Apr 3 at 4:48

Sehee ParkSehee Park

11

share|improve this question

asked Apr 3 at 4:48

Sehee ParkSehee Park

11

asked Apr 3 at 4:48

Sehee ParkSehee Park

11

asked Apr 3 at 4:48

Sehee ParkSehee Park

11