# Mann-Whitney U Test¶

Starting with some notes:

• Hypothesis testing is similar to predictive modeling in that you want to guard against both false positives and false negatives.
• For any given sample size, the t-test will usually be more robust than Mann-Whitney IF the distributions are normal.
• The t-test will be comparing Means and assumes that both distributions are similar (normal, to be specific).
• If the distributions are not normal, the Mann-Whitney U test will almost always be more reliable. (That's good since it was designed for this sort of case.)
• If the two distributions have the same shape then MW is comparing the Medians. If the distros have different shape, then you are technically comparing Mean Ranks. That is likely OK, but it depends on the whole situation. See this link for more info.

References: Stats Theory Theory Code

In [ ]:
# PRE-REQUISITS
import seaborn as sns
import numpy as np
import pandas as pd
%matplotlib inline

In [ ]:
# GET DATA

# empty df
df = pd.DataFrame()
# Get the global dataset, already cleansed and incl. features.
%run 'returnCleansedDatasetGlobal.py'

In [ ]:
data = df.copy()  # creating a safety copy
# df = df.where(df['average_spending'] < 200000)  # want to be looking only at "smaller" deals which make up the biggest bunch of our data?

# Taking a first look on the distribution of consulting customers vs. non_consulting customers
sns.displot(df, x="average_spending", bins=40, hue='consulting_customer')


# A/B Testing¶

### Assumptions:¶

• Check normality
• If Normal Distribution, check homogeneity

### Steps:¶

• Split & Define Control Group & Test Group
• Apply Shapiro Test for normality
• If parametric apply Levene Test for homogeneity of variances

### Action¶

• If Parametric + homogeneity of variances apply T-Test
• If Parametric - homogeneity of variances apply Welch Test
• If Non-parametric apply Mann Whitney U Test directly
In [ ]:
# Define A/B groups
ab = df.copy()
ab["group"] = np.where(ab.consulting_customer == True, "A", "B")

In [ ]:
# A/B Testing Function - Quick Solution
def AB_Test(dataframe, group, target):

# Packages
from scipy.stats import shapiro
import scipy.stats as stats

# Split A/B
groupA = dataframe[dataframe[group] == "A"][target]
groupB = dataframe[dataframe[group] == "B"][target]

# Assumption: Normality
ntA = shapiro(groupA) < 0.05
ntB = shapiro(groupB) < 0.05
# H0: Distribution is Normal! - False
# H1: Distribution is not Normal! - True

if (ntA == False) & (ntB == False): # "H0: Normal Distribution"
# Parametric Test
# Assumption: Homogeneity of variances
leveneTest = stats.levene(groupA, groupB) < 0.05
# H0: Homogeneity: False
# H1: Heterogeneous: True

if leveneTest == False:
# Homogeneity
ttest = stats.ttest_ind(groupA, groupB, equal_var=True)
# H0: M1 == M2 - False
# H1: M1 != M2 - True
else:
# Heterogeneous
ttest = stats.ttest_ind(groupA, groupB, equal_var=False)
# H0: M1 == M2 - False
# H1: M1 != M2 - True
else:
# Non-Parametric Test
ttest = stats.mannwhitneyu(groupA, groupB)
# H0: M1 == M2 - False
# H1: M1 != M2 - True

# Result
temp = pd.DataFrame({
"AB Hypothesis":[ttest < 0.05],
"p-value":[ttest]
})
temp["Test Type"] = np.where((ntA == False) & (ntB == False), "Parametric", "Non-Parametric")
temp["AB Hypothesis"] = np.where(temp["AB Hypothesis"] == False, "Fail to Reject H0", "Reject H0")
temp["Comment"] = np.where(temp["AB Hypothesis"] == "Fail to Reject H0", "A/B groups are similar!", "A/B groups are not similar!")

# Columns
if (ntA == False) & (ntB == False):
temp["Homogeneity"] = np.where(leveneTest == False, "Yes", "No")
temp = temp[["Test Type", "Homogeneity","AB Hypothesis", "p-value", "Comment"]]
else:
temp = temp[["Test Type","AB Hypothesis", "p-value", "Comment"]]

# Print Hypothesis
print("# A/B Testing Hypothesis")
print("H0: A == B")
print("H1: A != B", "\n")

return temp


In [ ]:
# Apply A/B Testing
AB_Test(dataframe=ab, group = "group", target = "average_spending")

# A/B Testing Hypothesis
H0: A == B
H1: A != B


Out[ ]:
Test Type AB Hypothesis p-value Comment
0 Non-Parametric Reject H0 3.499396e-54 A/B groups are not similar!
In [ ]:
# Apply A/B Testing
AB_Test(dataframe=ab, group = "group", target = "number_of_solutions")

# A/B Testing Hypothesis
H0: A == B
H1: A != B


Out[ ]:
Test Type AB Hypothesis p-value Comment
0 Non-Parametric Reject H0 1.248745e-91 A/B groups are not similar!