Data Science How-To: Using Apache Spark for Sports Analytics

Apache Spark has become a common tool in the data scientist’s toolbox, and in this post we show how to use the recently released Spark 2.1 for data analysis using data from the National Basketball Association (NBA). All code and examples from this blog post are available on GitHub.

Analytics have become a major tool in the sports world, and in the NBA in particular analytics have shaped how the sport is played. The league has skewed towards taking more 3-point shots due to their high efficiency as measured by points per field goal attempt. In this post we evaluate and analyze this trend in the NBA using season statistics data going back to 1979 along with geospatial shot chart data. The concepts in this post — data cleansing, visualization, and modeling in Spark — are general data science concepts and are applicable for other tasks beyond analyzing sports data. The post concludes with the author’s general impressions about using Spark and with tips and suggestions for new users.

For the analyses, we use Python 3 with the Spark Python API (PySpark) to create and analyze Spark DataFrames. In addition, we utilize both the Spark DataFrame’s domain-specific language (DSL) and Spark SQL to cleanse and visualize the season data, finally building a simple linear regression model using the spark.ml package — Spark’s now primary machine learning API.

Finally, we note that the analysis in this tutorial can be run with a distributed Spark setup running on a cloud service such as Amazon Web Service (AWS) or on a Spark instance running on a local machine. We have tested both and have included resources for getting started on either AWS or a local machine at the end of this post.

The Code

Using data from Basketball Reference, we read in the season total stats for every player since the 1979-80 season into a Spark DataFrame using PySpark. DataFrames are designed to ease processing large amounts of structured tabular data on the Spark infrastructure and are now in fact just a type alias for a Dataset of Row.

We can also view the column names of our DataFrame:

print(df.columns)

['_c0', 'player', 'pos', 'age', 'team_id', 'g', 'gs', 'mp', 'fg', 'fga', 'fg_pct', 'fg3', 'fg3a', 'fg3_pct', 'fg2', 'fg2a', 'fg2_pct', 'efg_pct', 'ft', 'fta', 'ft_pct', 'orb', 'drb', 'trb', 'ast', 'stl', 'blk', 'tov', 'pf', 'pts', 'yr']

Using our DataFrame, we can view the top 10 players, sorted by number of points in an individual season. Notice we use the toPandas function to retrieve our results. The corresponding result looks cleaner for display than when using the take function.  

df.orderBy('pts',ascending = False).limit(10).toPandas()[['yr','player','age','pts','fg3']]

df.orderBy('game_date').limit(10).toPandas()[['yr','name','game_date','shot_distance','x','y','shot_made_flag']]

We can query an individual player and season and visualize their shots locations. We built a plotting function plot_shot_chart (see the GitHub repo) that is based on Savvas Tjortjoglou's example.

As an example, we query and visualize Steph Curry's 2015-16 historic shooting season using a hexbin plot, which is a two-dimensional histogram with hexagonal-shaped bins.

The shot chart data is rich in information, but it does not specify if the shot type is a 3-point attempt or a corner 3.  We solve for this by building User Defined Functions (UDF), which identify the shot type given the xy coordinates of the shot attempt.

Here we defined our shot labeling functions using standard Python functions utilizing numpy routines.

We then register our UDFs and apply each UDF to the entire dataset to classify each shot type:

We can visualize the change in the shot selection over the past 6 years using all of our data from the 2010-11 season up until the 2015-16 season. For visualization purposes, we exclude all shot attempts taken inside of 8 feet as we would like to focus on the midrange and 3 point shots.

Over the years, there is a notable trend towards more three-pointers and fewer midrange shots.

Finally, we evaluate shot efficiency as a function of shot distance.

We then plot our results.

Among the top scorers in the league, close 3-point attempts are among the most efficient shots in the league, on par with shots taken close to the basket. It's no wonder that accurate 3-point shooting is among the most coveted talents in the NBA today!

Conclusion

Lastly as a seasoned data scientist, SQL user, and Python junkie, here are my two cents on getting started with Spark. The Spark ecosystem and documentation are continually evolving and it is important to use the newest Spark version. A first time user will notice there are multiple ways to solve a problem using different languages (Scala, Java, Python, R), different APIs (Resilient Distributed Dataset (RDD), Dataset, DataFrame), and different data manipulation routines (DataFrame DSL, Spark SQL). Many choices are up to the users and others are guided by the documentation. Since Spark 2.0 for example, the DataFrame is now the primary Spark API for Python and R users (rather than the original and still useful RDD). In addition, the DataFrame-based spark.ml package is now the primary machine learning API in Spark replacing the RDD-based API. Bottom line: the platform is evolving and it pays to stay up to date.

In this post, we’ve demonstrated how to use Apache Spark to accomplish key data science tasks including data exploration, visualization, and model building. These principles are applicable to other data science tasks and datasets, and we encourage you to check out the repository and try it on your own!

Additional Resources

• Install Spark on a local machine (Mac OS X Yosemite)

• AWS and PySpark with Anacona – Quick Start

• Install Spark on AWS