Categorical Variable Algorithms - Class Notes
Clustering (Unsupervised Learning)¶
K - Means Algorithm¶
k = number of clusters The algorithm creates k centroids and calculates the means of the distances from the centroid to the distances of the data points and continues to do so until the calculated means do not change or the centroids do not switch between groups of data.
In [1]:
from sklearn.cluster import KMeans
from sklearn.metrics import silhouette_score
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
%matplotlib inline
In [2]:
data = pd.read_excel('cars.xls')
data.head()
Out[2]:
In [3]:
x = data[['Price','Mileage','Cylinder']]
x.head()
Out[3]:
In [4]:
model = KMeans(n_clusters = 5)
model
Out[4]:
In [5]:
model = model.fit(x)
In [6]:
pred = model.predict(x)
pred
Out[6]:
In [7]:
x['new cluster'] = pred
x.head()
Out[7]:
The silhouette value is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The silhouette ranges from -1 to 1, where a high value indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters. If most objects have a high value, then the clustering configuration is appropriate. If many points have a low or negative value, the the clustering configuration may have too many or too few clusters. The Silhouette Coefficient for a sample is (b-a)/max(a,b)
In [8]:
silhouette_score(x,pred)
Out[8]:
In [9]:
# Visualize the data clusters
plt.figure(figsize=(10,6))
plt.scatter(x.Mileage,x.Price,c=pred)
plt.legend()
plt.colorbar()
plt.show()
In [10]:
model.cluster_centers_
Out[10]:
In [11]:
model.labels_
Out[11]:
In [12]:
# Creating a loop to loop through a range of clusters and see which number of clusters gives us the best silhouette score,
# thus giving us the best model
silScores = []
def clust(clusters):
bestClustNum = 0
silScore = 0
for num_of_cluster in np.arange(1,clusters):
num_of_cluster += 1
model = KMeans(n_clusters=num_of_cluster)
model = model.fit(x)
pred = model.predict(x)
score = silhouette_score(x,pred)
print 'Number of Clusters {}, silhouette score {}'.format(num_of_cluster,score)
if score > silScore:
silScore = score
bestClustNum = num_of_cluster
silScores.append(score)
print 'The best number of Clusters is {} with a silhouette score of {}'.format(bestClustNum,silScore)
clust(20)
silScores
Out[12]:
Classification Algorithm¶
k nearest neighbors calculates distance from points to groups of the points closest to it and determines which quality it is most like. It is highly recommended to scale or normalize your data when using K-nearest neighbors so some values do not have significantly higher weight than others.
In [13]:
data = pd.read_csv('heightweight.csv')
data_weight = data[['weightLb','ageYear']]
data_weight.head()
Out[13]:
In [14]:
from sklearn.preprocessing import scale
data_scaled = scale(data_weight)
data_scaled
Out[14]:
In [15]:
from sklearn.neighbors import KNeighborsClassifier
neigh = KNeighborsClassifier(n_neighbors=25)
model = neigh.fit(data_weight,data.sex)
In [16]:
print "Predicted Gender = ",model.predict([[110,19.0]])
Naive Bayes¶
Naive because we assume that all conditions are independent of each other. This assumption makes the math easier and the algorithm faster. However, this assumption is generally not true. Top uses for this algorithm are spam filters and document classification.
In [17]:
import os
import io
import numpy as np
from pandas import DataFrame
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
In [20]:
!cd
In [49]:
filenames = []
for files in os.walk('C:\\Users\\nwerner\\DevMasters\\Class Notes\\Day #5 - Classifier Algorithms'):
filenames.append(files)
In [50]:
filenames
Out[50]:
In [51]:
print os.path
In [52]:
loca = 'C:\\Users\\nwerner\\AppData\\Local\\Continuum\\Anaconda2\\lib\\ntpath.pyc'
In [53]:
print os.path.split(loca)
print os.path.splitdrive(loca)
print os.path.dirname(loca)
print os.path.basename(loca)
In [54]:
def readFiles(path):
for root, dirnames, filenames in os.walk(path):
for filename in filenames:
path = os.path.join(root, filename)
lines = []
f = io.open(path, 'r', encoding='latin1')
for line in f:
lines.append(line)
f.close()
message = '\n'.join(lines)
yield path, message
def dataFrameFromDirectory(path, classification):
rows = []
index = []
for filename, message in readFiles(path):
rows.append({'message': message, 'class': classification})
index.append(filename)
return DataFrame(rows, index=index)
In [56]:
data = DataFrame({'message':[],'class':[]})
data = data.append(dataFrameFromDirectory('C:\\Users\\nwerner\\DevMasters\\Class Notes\\Day #5 - Classifier Algorithms\\Emails\\spam','spam'))
data = data.append(dataFrameFromDirectory('C:\\Users\\nwerner\\DevMasters\\Class Notes\\Day #5 - Classifier Algorithms\\Emails\\ham','ham'))
data
Out[56]:
In [57]:
vectorizer = CountVectorizer()
# this will tokenize your data. It will take all message values and turn them into numeric values
# a word occurs and splits them into an array
counts = vectorizer.fit_transform(data['message'].values)
In [58]:
print counts
In [59]:
counts.todense()
Out[59]:
In [60]:
classifier = MultinomialNB()
# targets is tkhe classification of every email you have encountered
targets = data['class'].values
# this will create a model using naive bayes
classifier.fit(counts,targets)
Out[60]:
In [61]:
examples = ['Free Weed and Viagra now!!!','Mustang','Free Viagra']
example_counts = vectorizer.transform(examples)
predictions = classifier.predict(example_counts)
predictions
Out[61]:
Decision Trees¶
In [29]:
import numpy as np
import pandas as pd
from sklearn import tree
input_file = 'PastHires.csv'
df = pd.read_csv(input_file,header=0)
df.head()
Out[29]:
In [30]:
df.info()
In [31]:
d = {'Y':1,'N':0,'BS':0,'MS':1,'PhD':2}
df['Hired'] = df['Hired'].map(d)
df['Employed?'] = df['Employed?'].map(d)
df['Top-tier school'] = df['Top-tier school'].map(d)
df['Interned'] = df['Interned'].map(d)
df['Level of Education'] = df['Level of Education'].map(d)
df
Out[31]:
In [59]:
features = list(df.columns[:6])
y = df['Hired']
X = df[features]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(X,y)
clf
Out[59]:
In [60]:
print clf.predict([0,1,0,0,0,0]) # Just need a job to get hired
print clf.predict([0,0,0,0,0,1]) # Just need to have interned with the company to get hired
Ensemble Learning: Using a Random Forest¶
In [61]:
from sklearn.ensemble import RandomForestClassifier
# How many trees do you want in your forest?
clf = RandomForestClassifier(n_estimators=20)
clf = clf.fit(X,y)
clf
Out[61]:
Ensemble Learning: Using Gradient Boosting¶
In [63]:
from sklearn.ensemble import GradientBoostingClassifier
# How many trees do you want in your forest?
clf = GradientBoostingClassifier(n_estimators=10)
clf = clf.fit(X,y)
print clf.predict([10,1,4,0,0,0])
print clf.predict([10,0,4,2,0,0])
print clf.predict([0,1,0,0,0,0]) # Just need a job to get hired
print clf.predict([0,0,0,0,0,1]) # Just need to have interned with the company to get hired
In [65]:
prediction = clf.predict(X)
prediction
Out[65]:
In [66]:
from sklearn.metrics import accuracy_score, f1_score
accuracy_score(y,prediction)
Out[66]:
Logistic Regression¶
In [73]:
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
logmodel = lr.fit(X,y)
logmodel.predict([10,0,3,1,1,1])
logmodel.predict([45,0,1,2,1,1])
Out[73]:
In [ ]: