Trjtdtk

I recently came across the concept of embeddings so the concept is still new to me, but it is my understanding that embeddings convert one-hot encoded input data into a dense vector.

Vectors corresponding to all one-hot encodings are first embedded into the dense space randomly. As the embedding gets trained, the vectors move in the dense space from their initial random positions to positions that add algebraic meaning to the dense space. The data basically organizes itself.

This process of data arranging itself seems too good to be true.

I am posting this question here because in all the articles I have read about embeddings, they are presented as a silver bullet, and I wasn't able to find any texts on the limitations of embeddings.

This would essentially enable all categorical data to be converted into dense, meaningful representations during preprocessing or as part of training of a bigger model).

This is an example of embeddings being used to convert all categorical features into dense representations:

# Input layer for religion
encoder_liv = Sequential()
encoder_liv.add(Embedding(liv_cats,4,input_length=1))
encoder_liv.add(Flatten())

# Input layer for religion
encoder_edu = Sequential()
encoder_edu.add(Embedding(edu_cats,4,input_length=1))
encoder_edu.add(Flatten())

# Input layer for triggers(x_b)
dense_x = Sequential()
dense_x.add(Dense(4, input_dim=x.shape[1]))

model = Sequential()
model.add(Merge([encoder_liv, encoder_edu, dense_x], mode='concat'))
# model.add(Activation('relu'))
model.add(Dense(output_dim=12))
model.add(Activation('relu'))
model.add(Dense(output_dim=3))
model.add(Activation('softmax'))

model.compile(optimizer='adagrad', loss='categorical_crossentropy', metrics=['accuracy'])

An embedding is used in this video to replace the entire encoder network of a convolutional autoencoder generating human faces. here is the accompanying code.

So, what's the catch? Why should I not use always use an embedding for all categorical data? What are the constraints limiting an embedding's use/success?