Trjtdtk

Problem

I have data that includes multiple different text inputs as well as floats, categories, etc. Therefore I need to pass several different data types as features, including text which is an int array when tokenized.

Question

Say I tokenize the several text inputs; can I pass the tokenized text array as a feature alongside my floats and categories? If not, how is this done?

Background

When I've done NLP models, my code looks similar to this:

...
tokenizer = Tokenizer(num_words=max_features)
tokenizer.fit_on_texts(df['Stem'])
list_tokenized_train = tokenizer.texts_to_sequences(df['Stem'])
X_train = pad_sequences(list_tokenized_train, maxlen=10)
y_train = df['TotalPValue']
...

So, the text input becomes an array of int tokens padded with zeroes, e.g. [0 0 0 0 0 0 0 1 12 52].

This is not enough to solve my problem. I want to instead use multiple tokenized string inputs and floats as features. I want to first tokenize and pad each text input like above and put them in the same input array, like this: X_train = [[0 0 0 0 0 0 0 1 12 52], [0 0 0 0 0 0 0 42 12 23], 0.0425672].

I want to then start my model like this:

model = Sequential() 
model.add(Embedding(max_features, embedding_vector_length, input_length=3)) 

Will it work if implemented like this?

My attempts

I searched for a while but couldn't find anyone else doing it like this. Surprising to me that I couldn't find anything since it seems like a basic problem.

Just wanted to know if I have the right idea, since - as a beginner - implementation will cost a lot of time if this isn't the right way of doing it. Thanks so much for the insight!

Found a fantastic answer here that avoids high-level black box libraries.

Essentially, numerical floats are categorized into bins based on boundary values which are determined by their distribution. Text tokens are hashed with column ID, and then concatenated with other columns' hashed tokens via an interaction array. All hashed tokens, whether or not they have in interaction, are fed as inputs into the model.

That's the gist of it.