Dealing with
sparsity
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
Rob O'Callaghan
Director of Data
Sparse matrices
BUILDING RECOMMENDATION ENGINES IN PYTHON
Sparse matrices
BUILDING RECOMMENDATION ENGINES IN PYTHON
Sparse matrices
BUILDING RECOMMENDATION ENGINES IN PYTHON
Measuring sparsity
print(book_rating_df)
title The Great Gatsby The Catcher in the Rye Fifty Shades of Grey
User
User_233 3.0 NaN NaN
User_651 NaN 5.0 4.0
User_965 4.0 3.0 NaN
... ... ... ...
BUILDING RECOMMENDATION ENGINES IN PYTHON
Measuring sparsity
number_of_empty = book_ratings_df.isnull().values.sum()
total_number = user_ratings_df.size
sparsity = number_of_empty/total_number
print(sparsity)
0.0114
BUILDING RECOMMENDATION ENGINES IN PYTHON
Why sparsity matters
BUILDING RECOMMENDATION ENGINES IN PYTHON
Why sparsity matters
BUILDING RECOMMENDATION ENGINES IN PYTHON
Why sparsity matters
BUILDING RECOMMENDATION ENGINES IN PYTHON
Why sparsity matters
BUILDING RECOMMENDATION ENGINES IN PYTHON
Measuring sparsity per column
user_ratings_df.notnull().sum()
The Pelican Brief 1
Snow Crash 1
The Great Gatsby 12
Fifty Shades of Grey 9
Leviathan 1
..
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix factorization
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix factorization
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix factorization
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix multiplication
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix multiplication
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix multiplication
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix multiplication
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix multiplication
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix multiplication
print(matrix_x)
[[4, 1],
[2, 2],
[3, 3]]
print(matrix_b)
[[1, 0, 4],
[0, 1, 6]]
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix multiplication
import numpy as np
dot_product = np.dot(matrix_x, matrix_b)
print(dot_product)
[[ 4 1 22]
[ 2 2 20]
[ 3 3 30]]
BUILDING RECOMMENDATION ENGINES IN PYTHON
Let's practice!
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
Matrix factorization
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
Rob O'Callaghan
Director of Data
Why this helps with sparse matrices
BUILDING RECOMMENDATION ENGINES IN PYTHON
Why this helps with sparse matrices
BUILDING RECOMMENDATION ENGINES IN PYTHON
Why this helps with sparse matrices
BUILDING RECOMMENDATION ENGINES IN PYTHON
What matrix factorization looks like
BUILDING RECOMMENDATION ENGINES IN PYTHON
What matrix factorization looks like
BUILDING RECOMMENDATION ENGINES IN PYTHON
What matrix factorization looks like
BUILDING RECOMMENDATION ENGINES IN PYTHON
What matrix factorization looks like
BUILDING RECOMMENDATION ENGINES IN PYTHON
Latent features
BUILDING RECOMMENDATION ENGINES IN PYTHON
Latent features
BUILDING RECOMMENDATION ENGINES IN PYTHON
Latent features
BUILDING RECOMMENDATION ENGINES IN PYTHON
Information loss
BUILDING RECOMMENDATION ENGINES IN PYTHON
Information loss
BUILDING RECOMMENDATION ENGINES IN PYTHON
Information loss
BUILDING RECOMMENDATION ENGINES IN PYTHON
Information loss
BUILDING RECOMMENDATION ENGINES IN PYTHON
Let's practice!
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
Singular value
decomposition
(SVD)
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
Rob O'Callaghan
Director of Data
What SVD does
BUILDING RECOMMENDATION ENGINES IN PYTHON
What SVD does
BUILDING RECOMMENDATION ENGINES IN PYTHON
What SVD does
BUILDING RECOMMENDATION ENGINES IN PYTHON
What SVD does
BUILDING RECOMMENDATION ENGINES IN PYTHON
Prepping our data
print(book_ratings_df.shape)
(220, 500)
avg_ratings = book_ratings_df.mean(axis=1)
print(avg_ratings)
array([[4.5 ],
[3.5],
[2.5],
[3.5],
...
[2.2]])
BUILDING RECOMMENDATION ENGINES IN PYTHON
Prepping our data
user_ratings_pivot_centered = user_ratings_df.sub(avg_ratings, axis=0)
user_ratings_df.fillna(0, inplace=True)
print(user_ratings_df)
The Great Gatsby The Catcher in the Rye Fifty Shades of Grey
User_233 0.0 0.0 0.0
User_651 0.0 0.5 -0.5
User_965 0.5 -0.5 0.0
... ... ... ...
BUILDING RECOMMENDATION ENGINES IN PYTHON
Applying SVD
from scipy.sparse.linalg import svds
U, sigma, Vt = svds(user_ratings_pivot_centered)
print(U.shape)
(610, 6)
print(Vt.shape)
(6, 1000)
BUILDING RECOMMENDATION ENGINES IN PYTHON
Applying SVD
print(sigma)
[3.0, 4.8, -12.6, -3.8, 8.2, 7.3]
sigma = np.diag(sigma)
print(sigma)
array([ 3.0 , 0. , 0. , 0. , 0. , 0. ],
[ 0. , 4.8 , 0. , 0. , 0. , 0. ],
[ 0. , 0. , -12.6 , 0. , 0. , 0. ],
[ 0. , 0. , 0. , -3.8 , 0. , 0. ],
[ 0. , 0. , 0. , 0. , 8.2 , 0. ],
[ 0. , 0. , 0. , 0. , 0. , 7.3 ]),
BUILDING RECOMMENDATION ENGINES IN PYTHON
Getting the final matrix
BUILDING RECOMMENDATION ENGINES IN PYTHON
Getting the final matrix
BUILDING RECOMMENDATION ENGINES IN PYTHON
Getting the final matrix
BUILDING RECOMMENDATION ENGINES IN PYTHON
Getting the final matrix
BUILDING RECOMMENDATION ENGINES IN PYTHON
Calculating the product in Python
recalculated_ratings = np.dot(U, sigma)
BUILDING RECOMMENDATION ENGINES IN PYTHON
Calculating the product in Python
recalculated_ratings = np.dot(np.dot(U, sigma), Vt)
print(recalculated_ratings)
[[ 0.1 -0.9 -3.6. ... ]
[ -2.3 0.5 -0.5 ... ]
[ 0.5 -0.5 2.0 ... ]
[ ... ... ... ... ]]
BUILDING RECOMMENDATION ENGINES IN PYTHON
Add averages back
recalculated_ratings = recalculated_ratings + avg_ratings.values.reshape(-1, 1)
print(recalculated_ratings)
[[ 4.6 3.6 0.9 ... ]
[ 1.8 4.0 3.0 ... ]
[ 3.0 2.0 4.5 ... ]
[ ... ... ... ... ]]
print(book_ratings_df)
[[ 5.0 4.0 NA ... ]
[ NA 4.0 3.0 ... ]
[ 3.0 2.0 NA ... ]
[ ... ... ... ... ]]
BUILDING RECOMMENDATION ENGINES IN PYTHON
Let's practice!
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
Validating your
predictions
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
Rob O'Callaghan
Director of Data
Hold-out sets
BUILDING RECOMMENDATION ENGINES IN PYTHON
Hold-out sets
BUILDING RECOMMENDATION ENGINES IN PYTHON
Hold-out sets
BUILDING RECOMMENDATION ENGINES IN PYTHON
Hold-out sets
BUILDING RECOMMENDATION ENGINES IN PYTHON
Hold-out sets
BUILDING RECOMMENDATION ENGINES IN PYTHON
Hold-out sets
BUILDING RECOMMENDATION ENGINES IN PYTHON
Separating the hold-out set
actual_values = act_ratings_df.iloc[:20, :100].values
act_ratings_df.iloc[:20, :100] = np.nan
Generate predictions as before.
predicted_values = calc_pred_ratings_df.iloc[:20, :100].values
BUILDING RECOMMENDATION ENGINES IN PYTHON
Masking the hold-out set
mask = ~np.isnan(actual_values)
print(actual_values[mask])
[4. 4. 5. 3. 3. ...]
print(predicted_values[mask])
[3.76, 4.35, 4.95, 3.5869079 3.686337 ...]
BUILDING RECOMMENDATION ENGINES IN PYTHON
Introducing RMSE (root mean squared error)
BUILDING RECOMMENDATION ENGINES IN PYTHON
Introducing RMSE (root mean squared error)
BUILDING RECOMMENDATION ENGINES IN PYTHON
Introducing RMSE (root mean squared error)
BUILDING RECOMMENDATION ENGINES IN PYTHON
Introducing RMSE (root mean squared error)
BUILDING RECOMMENDATION ENGINES IN PYTHON
Introducing RMSE (root mean squared error)
BUILDING RECOMMENDATION ENGINES IN PYTHON
Introducing RMSE (root mean squared error)
BUILDING RECOMMENDATION ENGINES IN PYTHON
RMSE in Python
from sklearn.metrics import mean_squared_error
print(mean_squared_error(actual_values[mask],
predicted_values[mask],
squared=False))
3.6223997
BUILDING RECOMMENDATION ENGINES IN PYTHON
Let's practice!
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
Wrap up
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
Rob O'Callaghan
Director of Data
Non-personalized models
BUILDING RECOMMENDATION ENGINES IN PYTHON
Content-based models
BUILDING RECOMMENDATION ENGINES IN PYTHON
Collaborative filtering
BUILDING RECOMMENDATION ENGINES IN PYTHON
Matrix factorization
BUILDING RECOMMENDATION ENGINES IN PYTHON
Congratulations!
B U I L D I N G R E C O M M E N D AT I O N E N G I N E S I N P Y T H O N
