Trjtdtk

Airbnb - host wants to reduce rooms, can we get refund?

In gnome-terminal only 2 out of 3 zoom keys work

Find the coordinate of two line segments that are perpendicular

How can I get precisely a certain cubic cm by changing the following factors?

Stark VS Thanos

Was it really necessary for the Lunar Module to have 2 stages?

Any examples of headwear for races with animal ears?

What does 「再々起」mean?

How to set the font color of quantity objects (Version 11.3 vs version 12)?

Sci-fi novel series with instant travel between planets through gates. A river runs through the gates

Pulling the rope with one hand is as heavy as with two hands?

If Earth is tilted, why is Polaris always above the same spot?

What's the polite way to say "I need to urinate"?

Pawn Sacrifice Justification

Is creating your own "experiment" considered cheating during a physics exam?

Single Colour Mastermind Problem

Is it possible to measure lightning discharges as Nikola Tesla?

Will tsunami waves travel forever if there was no land?

Transfer over $10k

Given what happens in Endgame, why doesn't Dormammu come back to attack the universe?

In the time of the mishna, were there Jewish cities without courts?

Do I have an "anti-research" personality?

Can fracking help reduce CO2?

How to determine the actual or "true" resolution of a digital photograph?

How to handle columns with categorical data and many unique values

decision trees on mix of categorical and real value parametersPandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)Imputation of missing values and dealing with categorical valuesHow to deal with categorical variablesOne hot encoding error “sort.list(y)…”One hot encoding vs Word embeddingHow to implement feature selection for categorical variables (especially with many categories)?ML Models: How to handle categorical feature with over 1000 unique values“Binary Encoding” in “Decision Tree” / “Random Forest” AlgorithmsDealing with multiple distinct-value categorical variables

4

1

$begingroup$

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

machine-learning data categorical-data encoding

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

395

$endgroup$

add a comment | 

4

1

$begingroup$

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

machine-learning data categorical-data encoding

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

395

$endgroup$

add a comment | 

$begingroup$

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world.

I also have another column with 145 nunique values that I could also use in my model that represents product category.

Can I use one hot encoding to these columns or there's a problem with that solution?
Like which is the max number of unique values to use one hot encoding so there's not gonna be any problem ?

Can you point me to the right direction if I should use another encoding also?

machine-learning data categorical-data encoding

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

395

$endgroup$

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

395

share|improve this question

asked Apr 8 at 11:04

dungeondungeon

395

asked Apr 8 at 11:04

dungeondungeon

395

asked Apr 8 at 11:04

dungeondungeon

395

1 Answer
1

active

oldest

votes

5

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,6891414

$endgroup$

add a comment | 

Your Answer

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "557"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48875%2fhow-to-handle-columns-with-categorical-data-and-many-unique-values%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

5

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,6891414

$endgroup$

add a comment | 

5

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,6891414

$endgroup$

add a comment | 

$begingroup$

For categorical columns, you have two options :

1. Entity Embeddings


2. One Hot Vector

For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features.

Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one aspect (size) and very different in other aspects. So, instead of using ~3k column of features, model will have ~50 columns of vector representation.

Articles that explain Embeddings :

• An Overview of Categorical Input Handling for Neural Networks




• On learning embeddings for categorical data using Keras




• Google Developers > Machine Learning > Embeddings: Categorical Input Data




• Exploring Embeddings for Categorical Variables with Keras by Florian Teschner

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,6891414

$endgroup$

share|improve this answer

edited Apr 8 at 15:10

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,6891414

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,6891414

answered Apr 8 at 12:05

Shamit VermaShamit Verma

1,6891414