Trjtdtk

I am working on text classification where I have 39 categories/classes and 8.5 million records. (In future data and categories will increase).

Structure or format of my data is as follows.

----------------------------------------------------------------------------------------
| product_title | Key_value_pairs | taxonomy_id |
----------------------------------------------------------------------------------------
 Samsung S7 Edge | Color:black,Display Size:5.5 inch,Internal | 211 
 Storage:128 GB, RAM:4 GB,Primary Camera:12 MP 

 Case cover Honor 8 | Color:transparent,Height:15 mm,width:22 mm | 212 

 Ruggers Men's T-Shirt | Size:L,ideal for:men,fit:regular, | 111
 sleeve:half sleeve

 Optimum Nutrition Gold | Flavor:chocolate,form:powder,size:34 gm | 311
 Standard Whey Protein 

Data distribution is not normal; it is highly imbalanced:

-------------------------
| taxonomy_id | count |
-------------------------
 111 | 851750 
 112 | 355592
 113 | 379433
 114 | 23138
 115 | 117735
 116 | 145757
 117 | 1339471
 121 | 394026
 122 | 193433
 123 | 78299
 124 | 111962
 131 | 1776
 132 | 4425
 133 | 908
 134 | 23062
 141 | 22713
 142 | 42073
 211 | 7892
 212 | 1574744
 221 | 1047
 222 | 397515
 223 | 53009
 231 | 1227
 232 | 7683
 251 | 739
 252 | 327
 253 | 38974
 254 | 25
 311 | 2901
 321 | 7126
 412 | 856
 421 | 697802
 422 | 414855
 423 | 17750
 425 | 1240
 427 | 658
 429 | 1058
 431 | 20760
 441 | 257 

As you can see they are highly imbalanced and leading to mis-classifications.

Steps I have performed till now

1) Merge product_title and key_value_pairs column and remove stop words and special characters and perform stemming.

2) I have used pipeline for TFIDFvectorizer(), LinearSVC()

vectorizerPipe = Pipeline([
 ('tfidf', TfidfVectorizer(lowercase=True, stop_words='english')),
 ('classification', OneVsRestClassifier(LinearSVC(penalty='l2', loss='hinge'))),
 ])

After this I have fit pipeline and stored the classifier in pickle

prd = vectorizerPipe.fit(df.loc[:, 'description'], df.loc[:, 'taxonomy_id'])

On Testing side I have repeated step 1 as mentioned above and then load the pickle and use predict function

pd = cl.predict([testData])

Issues I am facing

1. 
   

   A lot of products are being mis-classified into some other categories

   
   
   

   Example: Ultimate Nutrition Prostar 100% Whey Protein should be classified into category 311 but my classifier is classifying it as 222 which is completely wrong.

   
   



2. I am not sure whether to use TFidfVectorizer() or Hashingvectorizer(), can you guys help me in selecting one of this along with their parameters?




3. Algorithm I am using is LinearSVC, is it a good choice for multi-class classification problems with large amount of data? Or should I use different algorithms?



4. 
   

   As my data is highly imbalanced I tried random undersampling. The results were improved but they were still not up to the mark.
   Also I am not sure whether this is the right approach to perform random undersampling:

   
   
   

   pipe = make_pipeline_imb(
    HashingVectorizer(lowercase=True),
    RandomUnderSampler(ratio=111: 405805, 112: 170431, 113: 241709, 114: 8341, 115: 50328, 116: 89445, 117: 650020, 121: 320803, 122: 162557, 123: 66156, 124: 36276, 131: 1196, 132: 3365, 133: 818, 134: 15001, 141: 6145, 142: 31783, 211: 24728, 212: 100000, 221: 791, 222: 8000, 223: 35406, 231: 785, 232: 3000, 251: 477, 252: 127, 253: 29563, 254: 33, 311: 2072, 321: 5370, 412: 652, 421: 520973, 422: 99171, 423: 16786, 425: 730, 427: 198, 429: 1249, 431: 13793, 441: 160,random_state=1), 
    OneVsRestClassifier(LinearSVC(penalty='l2', loss='hinge')))
   

   
   



5. I am new in machine learning so I have used this approach for text classification. If my approach is wrong then please correct me with right one.

(It would be great if you give suggestion or solution with examples as it will help me understand better).

***EDIT-1****

RndmFrst = RandomForestClassifier(n_estimators=100, max_depth=20, max_features=5000,n_jobs=-1)
LogReg = LogisticRegression()
voting = VotingClassifier(estimators=[('LogReg ', LogReg), ('RndmFrst', RndmFrst)], voting='soft', n_jobs=-1)

pipe = Pipeline([('tfidf', TfidfVectorizer(ngram_range=(1,4), max_features=50000)), ('clf', voting)])

pipe = pipe.fit(df.loc[:,'description'], df.loc[:,'taxonomy_id'])
Preds = pipe.predict(test_data)

Nice question!

Some Remarks

For imbalanced data you have different approaches. Most well-established one is resampling (Oversampling small classes /underssampling large classes). The other one is to make your classification hierarchical i.e. classify large classes against all others and then classify small classes in second step (classifiers are not supposed to be the same. Try model selection strategies to find the best).

Practical Answer

I have got acceptable results without resampling the data! So try it but later improve it using resampling methods (statistically they are kind of A MUST).

TFIDF is good for such a problem. Classifiers should be selected through model selection but my experience shows that Logistic Regression and Random Forest work well on this specific problem (however it's just a practical experience).

You may follow the code bellow as it worked simply well then you may try modifying it to improve your results:

train = pd.read_csv(...)
test = pd.read_csv(...) 

# TFIDF Bag Of Words Model For Text Curpos. Up to 4-grams and 50k Features
vec = TfidfVectorizer(ngram_range=(1,4), max_features=50000)
TrainX = vec.fit_transform(train)
TestX = vec.transform(test)


# Initializing Base Estimators
clf1 = LogisticRegression()
clf2 = RandomForestClassifier(n_estimators=100, max_depth=20, max_features=5000,n_jobs=-1)

# Soft Voting Classifier For Each Column
clf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2)], voting='soft', n_jobs=-1)
clf = clf.fit(TrainX, TrainY)
preds = clf.predict_proba(TestX)[:,1]

Please note that the code is abstract so TianX, TrainY,TestX,etc should be properly defined by you.

Hints

Be careful about what is StopWord. Practically many people (including myself!) made this mistake to remove stop words according to pre-defined lists. That is not right!

Stop words are corpus-sensitive so You need to remove stopwords according to information theoretic concepts (to keep it simple you need to know TFIDF kind of ignores your corpus-specific stopwords. If you need more explanation please let me know to update my answer).

VotingClassifier is a meta-learning strategy in the family of Ensemble Methods. They take benefit from different classifiers. Try them as they work pretty well in practice.

Voting schema simply takes the results of different classifiers and return the output of the one which has the highest probability to be right. So kind of democratic approach against dictatorship ;)

Hope it helps!