Trjtdtk

I am aware of the fact that the Pandas Dataframe's Statistical description can easily be obtained using df.describe(). I am having 2 dataframes of the same dimensions (i.e. 102 columns and 800000 rows for both the dataframes). I would like to depict the fact visually that the 2 dataframes are very similar/have a statistically similar distribution.

I am not sure of how this can be done visually/graphically in Python. The df.describe method provides count, mean, std, min, 25 %, 50 %, 75 % and max values for the dataframes, but it is difficult to infer from in the first instance. I would prefer a visual/graphical method of any type (I am not sure if Boxplot can help?). Any suggested method along with a Minimum Workable Example would be highly appreciated. Cheers!

Presently,

# df1 is dataframe 1
# df2 is dataframe 2
df1.describe() # this gives statistical values of df1 dataframe
df2.describe() # this gives statistical values of df2 dataframe

You can use the Kolmogorov-Smirnov Test. From Wikipedia

In statistics, the Kolmogorov–Smirnov test (K–S test or KS test) is a
nonparametric test of the equality of continuous, one-dimensional
probability distributions that can be used to compare a sample with a
reference probability distribution (one-sample K–S test), or to
compare two samples (two-sample K–S test). It is named after Andrey
Kolmogorov and Nikolai Smirnov.

Good for us, there is already an implementation for this in Scipy

You can use this dummy code to test it

from scipy import stats
p_value = 0.05
rejected = 0
for col in range(103):
 test = stats.ks_2samp(df1.ix[col,], df2.ix[col,])
 if test[1] < p_value:
 rejected += 1
print("We rejected",rejected,"columns in total")

I made the assumption that the columns that you want to compare are on the same index in both dataframes. If this is not true, you need to find another way. Maybe if they have the same name? You can do it like that.

from scipy import stats
p_value = 0.05
rejected = 0
for col in df1:
 test = stats.ks_2samp(df1[col], df2[col])
 if test[1] < p_value:
 rejected += 1
print("We rejected",rejected,"columns in total")

If the K-S statistic is small or the p-value is high, then we cannot reject the hypothesis that the distributions of the two samples are the same.

Edit: I haven't tried it before, but maybe you can do something like that

Boxplot

from scipy import stats
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

#create 2 dataframes with random integers. I don't have data to simulate your case.

df1 = pd.DataFrame(np.random.randint(0,100,size=(10000, 102)), columns=range(1,103))
df2 = pd.DataFrame(np.random.randint(0,100,size=(10000, 102)), columns=range(1,103))

#apply the Kolmogorov-Smirnov Test

p_value = 0.05
p_values = []
for col in range(103):
 test = stats.ks_2samp(df1.iloc[col,], df2.iloc[col,])
 p_values.append(test[1])

#create the box plot

plt.boxplot(p_values)
plt.title('Boxplot of p-values')
plt.ylabel("p_values")
plt.show()

Heatmap

Another way is the heatmap.

import matplotlib.patches as mpatches

plt.rcParams["figure.figsize"] = 5,2
x = range(1,104)
y = np.array(p_values)
fig, (ax,ax2) = plt.subplots(nrows=2, sharex=True)

extent = [x[0]-(x[1]-x[0])/2., x[-1]+(x[1]-x[0])/2.,0,1]
im = ax.imshow(y[np.newaxis,:], cmap="plasma", aspect="auto", extent=extent)
ax.set_yticks([])
ax.set_xlim(extent[0], extent[1])

values = np.unique(np.round(y.ravel(),2))

colors = [ im.cmap(im.norm(value)) for value in values]
# create a patch (proxy artist) for every color 
patches = [ mpatches.Patch(color=colors[i], label="Level l".format(l=values[i]) ) for i in range(len(values)) ]
# put those patched as legend-handles into the legend
plt.legend(handles=patches, bbox_to_anchor=(1.05, 2.2), loc=2, borderaxespad=0. )

ax2.plot(x,p_values)

plt.tight_layout()
plt.show()

Code for heatmap: stackoverflow