Trjtdtk

Having an imbalanced dataset. Abnormal class rate is %5. To handle with the problem I have gave extra weight to the abnormal class. However, It did not change anything. Here is my code:

from keras.models import Sequential
from keras.layers.core import Dense, Activation
import pandas as pd
import io
import requests
import numpy as np
from sklearn import metrics
import os
from sklearn.model_selection import train_test_split
from keras.models import Sequential
from keras.layers import Activation, Dense, Dropout, BatchNormalization
from keras.callbacks import EarlyStopping
from keras.utils import to_categorical
from keras.callbacks import ModelCheckpoint
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
from sklearn.utils import class_weight
from keras import optimizers
from keras.layers import Dropout
from sklearn.preprocessing import normalize
from sklearn.preprocessing import StandardScaler
from keras import regularizers
from sklearn.utils.class_weight import compute_sample_weight


def GenerateData(w,t,normal_size,abnormal_size):
#w: window length
#t: parameter of abnormal pattern (t=0.6/seperable, t=0.06/partially seperable, t=0.006/inseperable)
 data1=[]
 data2=[]
 mu, sigma = 0, 1

 for i in range(normal_size):
 x=np.random.normal(mu, sigma, w)
 data1.append(x)

 for i in range(abnormal_size):
 y=np.random.normal(mu, sigma, w)+t*(np.arange(w)+1)
 data2.append(y)


 data1=np.array(data1)
 data2=np.array(data2)


 data=np.concatenate((data1, data2), axis=0)

 labels=np.concatenate((np.ones(normal_size),np.zeros(abnormal_size)),axis=0)
 labels=labels.reshape(-1,1)

 Final_Data=np.concatenate((data, labels), axis=1)
 return Final_Data

Final_Data=GenerateData(20,0.06,950,50)
df=pd.DataFrame(Final_Data)

df = df.sample(frac=1).reset_index(drop=True)

X=df.iloc[:,:-1]
y=df.iloc[:,-1]
y = to_categorical(y)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)


scaler = StandardScaler()
X_train = scaler.fit_transform( X_train )
X_test = scaler.transform( X_test )

class_weight = class_weight.compute_class_weight('balanced', np.unique(y[:,-1]),y[:,-1])
#sample_weight = compute_sample_weight(class_weight='balanced', y=y_train)

model = Sequential()
model.add(Dense(8, input_dim=X_train.shape[1], activation='relu'))
model.add(Dense(y_train.shape[1],activation='softmax'))
opt=optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=1e-3, amsgrad=False)
model.compile(loss='categorical_crossentropy', optimizer=opt)
monitor = EarlyStopping(monitor='val_loss', min_delta=1e-8, patience=20, verbose=1, mode='auto')
checkpointer = ModelCheckpoint(filepath="best_weights.hdf5", verbose=0, save_best_only=True)
history=model.fit(X_train, y_train,validation_data=(X_test, y_test),verbose=2,class_weight=class_weight,callbacks=[monitor,checkpointer],epochs=2000)#classes are weighted
#history=model.fit(X_train, y_train,validation_data=(X_test, y_test),verbose=2,sample_weight=sample_weight,callbacks=[monitor,checkpointer],epochs=2000)# samples are weighted
#history=model.fit(X_train, y_train,validation_data=(X_test, y_test),verbose=2,callbacks=[monitor,checkpointer],epochs=2000)# no weighting

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()


model.load_weights('best_weights.hdf5') # load weights from best model


# Calculate accuracy
pred = model.predict(X_test)
pred = np.argmax(pred,axis=1)

y_compare = np.argmax(y_test,axis=1) 
score = metrics.accuracy_score(y_compare, pred)
print("Accuracy score: ".format(score))

cnf_matrix = confusion_matrix(y_compare, pred)

Based on the class_weight function, class weights are 10 and 0.52 for the abnormal and normal class respectively.
Whether given different weight or not did not change the performance of the model. Moreover, I have tried to give much more weight (1e+6) to abnormal class, but nothing changed. Model is not able to learn.

Instead of class_weight method, I have tried compute_sample_weight, but nothing changed.

So, what I am doing wrong or why the weighting strategy is not working properly in my case.

Although giving extra weight for handling imbalanced data-set is suggested, it's not a good way. I suggest you use an appropriate loss function for handling imbalanced data-set instead of giving weight to the abnormal class.

There are many useful metrics which were introduced for evaluating the performance of classification methods for imbalanced data-sets. Some of them are Kappa, CEN, MCEN, MCC, and DP.

Disclaimer:

If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix="Class1": "Class1": 1, "Class2":2, "Class2": "Class1": 0, "Class2": 5) 

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict #the prediction of the implemented model

>>> y_actu = data.target #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa #or any other parameter (Example: cm.SOA1)

Note that there is a large fluctuation in the errors after each run.

I have changed

class_weight = class_weight.compute_class_weight('balanced', np.unique(y[:,-1]),y[:,-1])

to

class_weight = np.array([1000, 1])

which resulted in val_loss around (0.09, 0.15)

and to

class_weight = np.array([1, 1000])

which resulted in val_loss around (0.06, 0.1)

So class weighting is working correctly and has an effect on the final result, but fluctuation is high. It is better to take an average on multiple runs. The negligible difference in test error simply means that weighting is not that important for this particular task.