CS 696 Intro to Big Data: Tools and Methods
Spring Semester, 2020
Doc 1 Introduction
Jan 23, 2020
Copyright ©, All rights reserved. 2020 SDSU & Roger Whitney, 5500
Campanile Drive, San Diego, CA 92182-7700 USA. OpenContent (http://
www.opencontent.org/openpub/) license defines the copyright on this
document.
Course Issues
http://www.eli.sdsu.edu/courses/index.html
Waitlist
Course Web Site
Wiki
Course Recordings
Prerequisites
This room
Grading
Books
Spark & Related Tools
Data Science
2
Waitlist - How to get into a Class
Add yourself to the course waitlist
Instructors can not
Add individuals to the class
See who is on the waitlist
Change your priority on the waitlist
3
Waitlist - How it works
Waitlist is a priority queue
When a seat in a class becomes available the top priority student is added
You can not be enrolled in two classes that meet at the same time
If wait list system adds you to a class, it will drop you from classes that meet at the same
time
First week of classes as students drop others are added
Second week of classes students are only added if instructor releases the seats
4
Can you add me to the Course?
Instructors can't select individual students to add to the course
5
Waitlist FAQ
Why not get a bigger room and admit everyone?
No first hard assignment to scare people
No Grader
Do you really want a 600 level class of 100 people?
This is the largest room of its type on campus
6
Waitlist FAQ
Will you be increasing the size of the class?
No
Why not?
No grader
New courses are a lot of work
Technology courses are a lot of work
7
Waitlist FAQ
Feb 4
Last day for regular students to add/drop classes
Open University students have lower priority than SDSU students
8
Waitlist FAQ
So what are my chances of adding this class?
Look up your position on the waitlist
What are the odds of that many people dropping the class
I can not see the waitlist
I have no idea how many people will drop
9
Grading
1 exam
4-6 assignments
Project
10
Course Website Demo
11
What are the Tools & Methods?
Programming language - Python
Programming Notebook
Visualization
scatter, box, violin, qq, line, density plots
errorbar, histogram, beeswarms
Statistics
mean, variance, quantiles, distributions
confidence intervals, correlation, coveriance
regression, goodness-of-fit, chi-squared test
Bayes theorem
Machine Learning
k-means, DBSCAN, Decision & Regression trees
Streaming - Kafka
Database - Cassandra
Hadoop, Spark, Pig, Mahout, etc.
12
What will be be doing
Installing programs
Python, Jupyter, Spark, Kafka, Cassandra
Writing Python, Java, Scala-Spark programs
Reports using Jupyter Notebooks
Analyzing data
Distributing data
Visualizing Data
Using Spark
Using Amazon Cloud
13
What will be be doing
~2 Weeks
Intro, Python
~5 weeks
Statistics, ML, NumPy, SciPy
Visualization
~3 weeks
Spark
~2 weeks
Kafka & Cassandra
14
Notebooks - Documentation, development
Python, Julia, R,
Other supported by community - Java, Fortran, Haskell, Ruby, Go, Scala, many more
Other notebook systems
Visualization
Python, Julia, R, Matlab
ML
Python (C), Julia, Matlab, R?
Spark - Large Data Sets Kafka - Streaming Data Cassandra - Data Storage
Scala Java Java
Java JVM languages Python
Python
Python Julia (Except for offsets) R - sort of
R Others - No R Client
Julia
15
Prerequisites
You will be installing software
Python
Jupyter
Some of these are more complex
Spark
on Windows than Unix/Mac OS
Kafka
Cassandra
Plotly
We will be doing some
Statistics
Math
Machine learning
16
Tasks - Install the Following
Jupyter via Anaconda & Conda with Python 3
http://jupyter.readthedocs.io/en/latest/install.html
Spark 2.4.4, Prebuild for Apache Hadoop 2.7
Unix/Linux/Mac OS
http://spark.apache.org/docs/latest/
Windows http://wiki.apache.org/hadoop/Hadoop2OnWindows
17
Books
Python Data Science Handbook: Essential Tools for Working with Data
Jake VanderPlas
O'Reilly Media
December 10, 2016
ISBN 9781491912058
Spark: The Definitive Guide
Matei Zaharia, Bill Chambers
February 2018
ISBN 9781491912218
18
Books
Course books are available for free on-line via SDSU library
Need SDSU Library account to access books off campus
Some people do not like reading books on-line
But if you need to save money it is available
May add chapters of other books as semester progresses
But on-line from books available on-line
19
Spark, Amazon
You will run Spark on Amazon’s cloud
You need to create an Amazon AWS account
Sign up for Amazon Educate account - $100 compute time for free
But you may incur some cost on Amazon
20
Data Science & Big Data
Very trendy
When topics become trendy in CS the terms become very vague
Big Data Analytics with Excel
Is Data Scientist A Useless Job Title?
21
Data Science
Data science is an interdisciplinary field about processes and systems to extract
knowledge or insights from data in various forms, either structured or
unstructured,[1][2] which is a continuation of some of the data analysis fields
such as statistics, data mining, and predictive analytics,[3] similar to Knowledge
Discovery in Databases (KDD)
Wikipedia
22
Data Science
Data Scientist (n.):
Person who is better at statistics than any software engineer and
better at software engineering than any statistician.
— Josh Wills (@josh_wills) May 3, 2012
23
Data Engineer
A software engineer that deals with data plumbing
Traditional database setup, Hadoop, Spark, etc.
Data analyst
A person who digs into data to surface insights,
but lacks the skills to do so at scale
They know how to use
Excel, Tableau and SQL
but can’t build a web app from scratch
24
Data Science
Science of transforming data into useful information by means of
Statistical and
Machine learning techniques
25
Data Science & Big Data
Big Data
Data Science with large datasets
No hard boundary between Big Data and medium data
Requires more data plumbing
26
Inconvenient Truth About Data Science
Data is never clean.
You will spend most of your time cleaning and preparing data.
95% of tasks do not require deep learning.
In 90% of cases generalized linear regression will do the trick.
Big Data is just a tool.
You should embrace the Bayesian approach.
No one cares how you did it.
Academia and business are two different worlds.
Presentation is key - be a master of Power Point.
All models are false, but some are useful.
There is no fully automated Data Science. You need to get your hands dirty.
27
Share of Respondents
0%
10%
20%
30%
40%
50%
60%
70%
SQL
Excel
Python
R
MySQL
Python: numpy, scipy, scikit-learn
ggplot
TOOLS
Microsoft SQL Server
Tableau
JavaScript
Matplotlib (Python)
Java
PostgreSQL
Oracle
D3
Homegrown analysis tools
Hive
28
Spark
Cloudera
Visual Basic/VBA
MongoDB
LANGUAGES, DATA PLATFORMS, ANALYTICS
Apache Hadoop
SAS
C++
PowerPivot
Tool: language, data platform, analytics
Scala
SQLite
C
Pig
Amazon RedShift
Weka
Hbase
Amazon Elastic MapReduce (EMR)
Perl
SPSS
Teradata
Share of Respondents
50K
100K
150K
200K
0K
SQL
Excel
Python
R
MySQL
Python: numpy, scipy, scikit-learn
ggplot
Microsoft SQL Server
Tableau
JavaScript
Matplotlib (Python)
SALARY MEDIAN AND IQR (US DOLLARS)
Java
PostgreSQL
Oracle
D3
Homegrown analysis tools
29
Hive
Spark
Cloudera
TOOLS: LANGUAGES, DATA PLATFORMS, ANALYTICS
Visual Basic/VBA
MongoDB
Tool: language, data platform, analytics
Apache Hadoop
SAS
C++
PowerPivot
Scala
SQLite
C
Pig
Amazon RedShift
Weka
Hbase
Amazon Elastic MapReduce (EMR)
Perl
SPSS
Teradata
30
Rule of Three
If you can not think of three things that might go wrong with your analysis
there is something wrong with your thinking
31
Data Science Verses Programming Jobs
Intuit Job Listing Worldwide Aug 22 2016
Data - 23
Software Engineer - 168
32
Data Science Programming Languages
Python Scala Java
R Julia C++
Matlab C
Javascript C#
SAS
Perl
Ruby
33
Features of Languages for Data Science
Interactive
Statistical, Machine Learning, Math libraries
Plays well with others
Supports computation
Simple syntax
Fast
34
Python
Wildly used Slow
Interactive Python 2.x verses Python 3.x
3/2
Lots of libraries
Threads do not scale
Plays well with other Global Interpreter Lock (GIL)
35
Julia
New language from MIT LLVM
Interactive & Fast Lisp style macros
Untyped & Typed Multiple dispatch
Designed for computation Designed for parallelism &
Distributed computation
f(x) = 2x + 4
Int32, Int64, Int128, BigInt
Statistical and Math libraries
Plays well with others
36
Java, Scala, Hadoop, Spark
Hadoop written in Java
Spark written in Scala
JVM languages (Java, Scala, Clojure, Groovy, JRuby, Jython)
Much more efficient on Hadoop & Spark
First access to new features
Scala
OO & Functional
Type inference
Far less verbose than Java
37
