Collaborative Filtering

Why not use SVD?

Answer: Too many missing entries, and imputation is expensive or inaccurate

Naive ALS

• R: ratings matrix
• U: user factors
• V: item factors

Problems?

Storing R

R is a very large matrix and possibly won’t fit in main memory

Sends duplicate copies to each worker

Distribute R

Store R as an RDD/DataFrame, but broadcast U and V

Problems?

Storing U and V

U and V might not fit in memory either

Sends duplicate copies to each worker

Join ALS

Store R, U, and V as an RDD/DataFrame

Blocked Join ALS

Spark implements a smarter version of join ALS to limit data shuffling

ALS is an example of a distributed model (i.e. stored across executors)