Confidential Customized for Lorem Ipsum LLC Version 1.

AI & ML Strategic
Frameworks for
Business Leaders
Technological Revolutions

Computers Smart Phone

Internet AI
Eventually AI will be the standard of doing business,
& AI will be everyday technology.
We’ll need not just Data Scientists
but Business Leaders
That Understand AI.

Also Known as Analytics Translators

Case Studies: Changing Role of AI for Business

“You Don’t Have to be a

Data Scientist to Fill this
Must-Have Analytics Role”
Harvard Business Review

“When AI Becomes
Everyday Technology”
Harvard Business Review
Confidential Customized for Lorem Ipsum LLC Version 1.0

1 2 3 4
Foundational Explore Business Review ML Project
Understanding of AI/ML Opportunities ML Options Management
Confidential Customized for Lorem Ipsum LLC Version 1.0

1
Foundational
Understanding of AI/ML
 1
AI & Machine
Learning 101
AI & Machine
Learning in
30 Seconds
When you hear about AI companies, what they
really are is a Machine Learning company.
Traditional Algorithms Machine Learning

Algorithms gives a machine instructions Machine Learning is showing a

on how to go from point A to point B. machine examples of going from
point A to B and having the machine
write instructions on how to do it.
Traditional Algorithms Machine Learning

50,000 lines of code 500 lines of code

Machine Learning in 3 Minutes

Machine Learning is about Finding Patterns

& then
Making Predictions based on those patterns
Machine Learning in 3 Minutes

+ = Learning
Machine Learning in 3 Minutes

Saw Patterns
& then
Made a Prediction
 Machine Learning in 3 Minutes

+ = Learning

If the animal has:

● Two ears
= Strange parenting practices
● A snout
● Walks on all 4s
● Black or Beige Nose
Machine Learning in 3 Minutes

+ = Training
Machine Learning in 3 Minutes

DOG Analyzed Patterns DOG

& then
Made a Prediction

DOG DOG DOG

Machine Learning in 3 Minutes

+ = ML Training
If the animal has:
= IF ELSE, Algorithms
● Two ears
● A snout
● Walks on all 4s
● Black or Beige Nose
Data Principles & the Importance of Data

Garbage Data In = Garbage Data Out

The Machine is only as good as the human data

Humans are still needed (to label data or validate)

AI, ML, DL
Do tasks normally only
humans do. [1950s]

Uses Statistics and

Algorithms. [1980s]

Uses Artificial Neural

Networks and
Algorithms. [2010s]
Do tasks normally only Artificial Intelligence
humans do. [1950s]

Natural Language
Computer Vision Processing (NLP)

Speech Robotics
Recognition
Artificial Intelligence

Artificial
Speculation - exceed human intelligence
SUPER example: the movie Ex Machina
Intelligence

Artificial Undergoing Research - match humans

GENERAL overall
Intelligence example: Alpha Go

Artificial
Achievable - match humans in one domain
NARROW example: speech recognition, image recognition
Intelligence
Uses Statistics and
Algorithms. [1980s]

(2)
(1)
+ =
MACHINE (3)
TRAINING
LEARNING MODEL
DATA
ALGORITHM
Machine Learning

Statistics
+ Big Data
+ Processing Power
+ Storage
= Classical Machine Learning
Uses Artificial Neural
Networks and
Algorithms. [2010s]
Deep Learning Neural Networks
Deep Learning Neural Networks
Black Box Problem & Feature Extraction
Traditional Machine Learning

Feature
Input Classification Output
Extraction

Deep Learning

Feature
Input + Classification Output
Extraction
Interpretability vs. Accuracy
 One is not necessarily better

Deep learning

Performance

Classical Machine
Learning

Amount of Data
 When to choose ML over DL

ML DL

Data Less Data More Data

Hardware Low End High End (GPU)

Training Hours to Days Days to Weeks

Input Human Feature Extraction Machine Feature Extraction

Testing Increases with Data Size Relatively Quick

Interpretability Easier Interpretability Black Box Problem

Workbook Exercise (5 minutes) Page 8

Choose ML or DL depending on the circumstance

Workbook Exercise (5 minutes) Page 8

● A photo sharing company with millions of user images that wants to train a
model to label what those images are.

● A bank wants to build AI that approves or denies an individual for a

mortgage.

● A student who wants to work on an AI project but has a short deadline to

build and then train the model.

● A business person that wants to work with Data Scientists to classify X-ray
scans but cannot find a radiologist to work with on building this model.

● A cash strapped AI team needs to build a model that can predict what
customers are most likely to churn.
Ways that a machine learns from data
for machine learning & deep learning

Supervised

Unsupervised

Reinforcement
Supervised Learning

Train the model on “labelled data” (known

outcomes) to predict on unseen data.

Show the machine what you want it to do.

Defaults on Does not Default

Loan on Loan
Supervised Learning

Supervised Learning

Regression Classification
(Quantity) (Category)

Neighbours
K-Nearest
Machine
Decision

Support
Vector
Linear

Ridge

Lasso

Trees
Unsupervised Learning

Try to uncover patterns within the data, with

no data labels provided.

Let the machine identify patterns of its own.

 K-Means

Principal
Component Analysis
Learning
Unsupervised

Collaborative
Filtering
Unsupervised Learning
Reinforcement Learning

Train the model based on goals and specific

parameters.

Tell the machine what the goal is,

and let it figure out how to achieve the goal.
Workbook Exercise (5 minutes) Page 10

Choose between supervised, unsupervised, or reinforcement learning

Workbook Exercise (5 minutes) Page 10

● Building a model that identifies pictures of cars

● Building a model that can play chess
● Building a model that identifies credit card fraud
● Building a model that identifies customer segments
● Building a model that predicts housing prices
● Building a model that can teach a robot to stock and pick inventory
Confidential Customized for Lorem Ipsum LLC Version 1.0

1
Foundational
Understanding of AI/ML
Confidential Customized for Lorem Ipsum LLC Version 1.0

1 2
Foundational Explore Business
Understanding of AI/ML Opportunities
 2
Exploring Business
Opportunities with AI
 Business Analysis

● SWOT
● AI Prioritization
● Hypothesis Building
● Business Canvas
SWOT Opportunity Analysis

Strengths Weaknesses

Opportunities Threats
Workbook Exercise (5 minutes) Page 12

SWOT Analysis
 1 Generate Insights
Four Types of AI
2 Create Operational Efficiencies
Business Model
3
Innovations Add to an Existing Product

4 Create a New Product

Workbook Exercise (10 minutes) Page 13

Idea generation for your business, or an imaginary business:

Ways ML could do one of the following:

1 Generate Insights

2 Create Operational Efficiencies

3 Add to an Existing Product

4 Create a New Product

Identifying Problems Worth Solving For

1) A human expert could perform the task in a few

minutes
2) It is difficult or impossible to write down all the rules
3) It’s easy to get examples of the desired behaviour

Look at your sticky notes. If you answered no all of the above

for one of the sticky note ideas, remove that sticky note.
 Always ask . . .
Do we really need AI to solve this problem?

Look at your sticky notes. If you answered no to any of the above,

remove the sticky note.
 AI Prioritization Graph
LOW HIGH

(Nice to Haves) (Should Haves)

Reinforcement Learning

HIGH
Natural Language
Processing

COMPLEXITY
Image/Sound Recognition

(Can Haves) (Must Haves)

Clustering/Customization LOW

Classification

Predictive Analytics
LOW HIGH

BENEFITS
Workbook Exercise (10 minutes) Page 15

AI Prioritization Graph
 With greater prediction capabilities,
Building an AI [company] will be capable of [Type of AI
Hypothesis Business Model Innovation] which will provide
the company with [expected benefit].
Workbook Exercise (5 minutes) Page 16

AI Hypothesis
1 Generate Insights
With greater prediction capabilities,
2 Create Operational Efficiencies
[company] will be capable of [Type
of AI Business Model Innovation] 3 Add to an Existing Product

which will provide the company with

4 Create a New Product
[expected benefit].
The AI
Business
Canvas
Workbook Exercise (after class) Page 17

Build your AI Business Canvas

 Test the Concept

Oz Testing

Or ask, if we could hire 1,000 more humans how much

would this benefit the business?
Case Studies: Business Applications

“Salesforce launches Service

Cloud Voice, Einstein Voice
Assistant, and Voice Skills”
Venture Beat

“The Amazing Ways TD

Bank, Uses Big Data, AI &
Machine Learning”
Forbes
Confidential Customized for Lorem Ipsum LLC Version 1.0

1 2
Foundational Explore Business
Understanding of AI/ML Opportunities
Confidential Customized for Lorem Ipsum LLC Version 1.0

1 2 3
Foundational Explore Business Review
Understanding of AI/ML Opportunities ML Options
 3
Review ML
Options
A B

The Data The Model

Do we have or can we get the data? Who will build the model?
A

The Data
Do we have or can we get the data?
A
Data Collection Types

1 Product Usage Data

Simple
2
Open Source Data

3
Humans Labeling Data
Difficult
4
New Features

5
Acquisition/Purchase
A

Product Usage Data

01
Database Data
Customer Resource Management
systems
Google Analytics Data
Email activity
A

Open Source Data

02
Kaggle
Common Voice (by Mozilla)
Government Open Data
Gutenberg Project (by Google)
etc.
A

Josh Garell, Netflix Tagger

Humans Labeling Data 40 Taggers Worldwide

03
Internal Hires
Hive.ai
Mechanical Turk
Cloud Factory
Figure Eight
A

Facebook’s Fixed Emoji Selection.

New Features Data Collection Mechanism.

04
New form fields
New functionality
New product options
A

Spotify Acquires Eco Nest

Acquisition/Purchase For 100 Million. 90% Equity.

05
Spotify acquired Eco Nest
Facebook acquired Face.com
Google acquired Waze
OR
Buy a dataset from organizations
Workbook Exercise (5 minutes) Page 19

Match data collection methods to problem statements

Workbook Exercise (5 minutes) Page 19

● A medium sized telecom business with voice recorded tech support calls,
wants to build out a voice authorization system using AI.

● A large e-commerce business wants to classify hundreds of thousands of

inventory pieces as what type of product it is, and has a large budget to do so.

● A news website wants AI to scan their ads for offensive material, but does not
have enough offensive images to train a model on.

● A large bank wants to leverage AI to help their customers make good

investment decisions but they don’t have data they need relating to
customer’s investment risk preferences.
 B

The Model
Who will build the model?
 B
ML Modeling Methods

1 In House
Expertis

2 Machine Learning as a Service

(MLaaS)
3 Enterprise AI (Third Party Vendors)
e
 B

In House
01
Staffing Needs

1. Data Scientist
2. Data Engineer
3. Product/Project Manager
4. Domain Expertise
 B

MLaaS
02
Also known as Machine Learning as a
Service
● Amazon Machine Learning
● Azure Machine Learning
● IBM Watson Machine Learning
● BigML

MLaaS providers offer tools including data

visualization, APIs, face recognition, natural
language processing, predictive analytics and deep
learning. The provider's data centers handle the
actual computation.
 B

Enterprise AI
03
When considering Enterprise AI look for:

1. Compatibility
2. Functionality
3. Price
4. Credibility
5. User Friendliness
6. Agility
7. Exit Strategy
 B

ML Modeling
Methods Decision In House

Making

MLaaS

Is this
problem
core to the
Enterprise AI
business?
 B

ML Modeling
Methods Decision In House

Making

MLaaS

Is this
problem
core to the
Enterprise AI
business?
No
 B

ML Modeling
Methods Decision In House

Making

MLaaS

Yes
Is this
problem
core to the
Enterprise AI
business?
No
 B

ML Modeling
Methods Decision In House

Making
Can you get
the data
needed? MLaaS

Yes
Is this
problem
core to the
Enterprise AI
business?
No
 B

ML Modeling
Methods Decision In House

Making
Can you get
the data
needed? MLaaS
No

Yes
Is this
problem
core to the
Enterprise AI
business?
No
 B

ML Modeling
Methods Decision In House

Making Yes
Can you get
the data
needed? MLaaS
No

Yes
Is this
problem
core to the
Enterprise AI
business?
No
 B

ML Modeling
Methods Decision In House
Do you have an
Making Yes
experienced
ML team?
Can you get
the data
needed? MLaaS
No

Yes
Is this
problem
core to the
Enterprise AI
business?
No
 B

ML Modeling
Methods Decision Yes In House
Do you have an
Making Yes
experienced
ML team?
Can you get
the data
needed? MLaaS
No

Yes
Is this
problem
core to the
Enterprise AI
business?
No
 B

ML Modeling
Methods Decision Yes In House
Do you have an
Making Yes
experienced
ML team?
Can you get No
the data
needed? MLaaS
No

Yes
Is this
problem
core to the
Enterprise AI
business?
No
Workbook Exercise (5 minutes) Page 21

Match modeling methods to problem statements

Workbook Exercise (5 minutes) Page 21

● A news website wants AI to scan their ads for offensive material, but does
not have enough offensive images to train a model on.

● A medium sized telecom business with voice recorded tech support calls,
wants to build out a voice authorization system using AI, and they have a
semi-experienced data science team to build this out.

● A small startup that builds home thermostat devices, has a large data
science team that can build Machine Learning models, and they are
interested in building a customer support chat bot that can more quickly
respond to customer complaints.

● A large bank wants to leverage AI to help their customers make good

investment decisions. They don’t have data they need relating to customer’s
investment risk preferences, but they do have an experienced data science
team that could build the model.
Workbook Exercise (5 minutes) Page 22

Identify the best Data Options for your ML Hypothesis

Identify the best ML Modeling Option for your ML Hypothesis

Return on
Investment
Net Profit
ROI =
Always remember that Total Investment
working with AI means
working with uncertainty
 SAVINGS: Consider the
REVENUE: Consider how cost of a human working
the new feature/insight on the problem.
might generate revenue. Assume machine is
only 70% as effective,
so some humans are
Return on still needed, however
the machine can work
at scale.
Investment
Net Profit
ROI =
Always remember that Total Investment
working with AI means
working with uncertainty

DATA & MODELING: x TIME: Remember AI

Include data & modeling takes much more time
costs whether it’s in that anticipated. Be very
house, our not. conservative
Workbook Exercise (5 minutes) Page 22

Return on Investment: statements (not numerical)

Confidential Customized for Lorem Ipsum LLC Version 1.0

1 2 3
Foundational Explore Business Review
Understanding of AI/ML Opportunities ML Options
Confidential Customized for Lorem Ipsum LLC Version 1.0

1 2 3 4
Foundational Explore Business Review ML Project
Understanding of AI/ML Opportunities ML Options Management
 4
ML Project
Management
ML FLYWHEEL

Andrew Ng
 ML Project Lifecycle

1. Data 2. Train 3. Test 5. Deploy 6. Manage

 Data Train Test Deploy Manage

Data

Pulling Data: Selecting Data: Preprocessing Data:

● Where does our data live ● What is available ● Formatting
● What is available ● What is missing ● Cleaning
● Where can we put the data ● What should be removed ● Sampling
needed ● What data might be biased ● Labeling
Data Train Test Deploy Manage

Data Scientist Training & Testing

Divide Ready Data into 3 Sets

~80% ~10% ~10%

Validation/
Training Data Development Data Testing Data

The sample of data used to fit the model. The data that is used The sample of data
The model learns from this data. to fine-tune the model used to provide an
hyperparameters. unbiased
Hence the model evaluation of a final
occasionally sees this model fit on the
data, but never does it training dataset.
learn from this data.

1 2 3
Data Train Test Deploy Manage

Non-Data Science Testing

Unsupervised Reinforcement
Hard to Validate Medium Validation
with no ground Difficulty.
truth.

Based on Future
Impact Test in the real
world environment
Against humans
Data Train Test Deploy Manage

Real World Tests

Reinforcement Learning
Data Train Test Deploy Manage

Non-Data Science Testing

Unsupervised Reinforcement Supervised

Hard to Validate Medium Validation Easy to Validate
with no ground Difficulty. with ground truth
truth. examples.

Regression Classification
Based on Impact
Test in the real
world environment
Against humans Actual Future Confusion
Outcomes Matrix
Data Train Test Deploy Manage

Confusion Matrix
Supervised Learning, Classifications
Data Train Test Deploy Manage

Confusion Matrix
Supervised Learning, Classifications

Actual Values
TRUE FALSE

TRUE
Predicted Values
True Positive False Positive
FALSE Correct Prediction Incorrect Prediction

False Negative True Negative

Incorrect Prediction Correct Prediction
 Data Train Test Deploy Manage

Accuracy
Percentage of classifications
that were correctly made.

True Positives + True Negatives

True Positives + True Negatives

+ False Positives + False Negatives
 Data Train Test Deploy Manage

Accuracy Precision
Percentage of classifications When the model classifies
that were correctly made. positively, how often is it right?
(Quality)

True Positives + True Negatives True Positives

True Positives + True Negatives True Positives + False Positives

+ False Positives + False Negatives
 Data Train Test Deploy Manage

Accuracy Precision Recall

Percentage of classifications When the model classifies How many positive classifications,
that were correctly made. positively, how often is it right? does the model capture?
(Quality) (Quantity)

True Positives + True Negatives True Positives True Positives

True Positives + True Negatives True Positives + False Positives True Positives + False Negatives
+ False Positives + False Negatives
Data Train Test Deploy Manage

Harmonic Mean/F1 Score

Summarizing Precision & Recall

Precision x Recall
F1 = 2 x
Precision + Recall
Workbook Exercise (5 minutes) Page 26

Actual Values
n=155
TRUE FALSE

Calculate the

TRUE
Precision, Recall

Predicted Values
True Positive False Positive
TP = 100 FP = 5
& F1 Score

FALSE
False Negative True Negative
FN = 30 TN = 20
Workbook Exercise (5 minutes) Page 27

Match User Sentiment from two businesses to the Confusion Matrix

Netflix wants AI to A hospital wants AI to sift through hundreds of

identify what movies mammograms to identify if a patient has
should be recommended breast cancer. If they are waved as having
to a user and which ones breast cancer they are brought in for further
should not. tests, if they are waved as not having breast
cancer they will not be given further testing.
Workbook Exercise (10 minutes) Page 28

Match User Sentiment from your own ML Hypothesis to the Matrix

Understand what you need to optimize for:

• Precision
• Recall
• F1
Data Train Test Deploy Manage

Deploy & Manage

1. Data Scientists are Not Data Engineers

Stop looking for Unicorn hires. Data Scientists work on the model,
Data Engineers work on the model productionisation.
 Data Train Test Deploy Manage

Deploy & Manage

2. Deployment does Not Always Mean Integration into Product

Especially with insights, you can merely export data insights.
 Data Train Test Deploy Manage

Deploy & Manage

3. Talk to Data Scientists in Advance about Deployment Possibilities

Some models are easier to deploy than others.
 Data Train Test Deploy Manage

Deploy & Manage

4. Upgrades in ML from more data not new code

Building AI Solutions is much more of a
Cross-Functional effort than people think.
COMPANY FOUNDATIONS

Machine
Learning

Data Science Team

Analytics Team

Big Data Culture

 FOUNDATIONAL ML TEAM

Data Scientist: Data Engineer: Project/Product Software Engineer:

Enables advanced Develops and Maintains Manager: [This hire depends on how
analysis and modeling the ingestion of data, Has a deep understanding the model will be
to extract meaning in architecture for data of the company and deployed: on Web?
data, find relationships, warehousing, and industry, and ensures AI Mobile? Internally? They
and make predictions. pipelines for reliable data. initiatives are aligned with will be needed to help
the business. build the interface
between the model and
the user.]
GROWING AN ML TEAM

1 Data Engineer can support

around 3 Data Scientists
 ML TEAM ENGAGEMENTS

Model
Data Sourcing Data Preparation Data Model
Build/Train
15% 25% Exploration Implementation
30%
10% 20%

Data Engineer Data Engineer

Data Scientist

Software Engineer
WORKING WITH AN ML TEAM

1. Treat your ML Project as a Partnership.

2. Provide a clear problem, hypothesis, and goal
3. Don’t try and hire “unicorn” Data Scientists
4. Science experimentation means uncertainty
The future of ML will require more
Analytics Translators like you.
Ethics
Case Studies: Data Biases
“An AI Run world needs to
better reflect people of
colour”
Wired

“Amazon scraps secret AI

recruiting tool that showed
bias against women”
Reuters
Discussion:

What data should Amazon used instead of hiring decisions

(yes/no) from humans, that could remove the bias?
Risks/Security
Data Flow
User

Operational/ Model & Service

Internal Data

Client/ AI Pipeline
Collected Data

External Data Model (endpoint

deployment)
Adversarial Samples
User

Operational/ Model & Service

Internal Data

Client/ AI Pipeline
Collected Data

External Data Model (endpoint

deployment)
Adversarial Samples
Adversarial Samples
Adversarial examples are inputs to a neural network that
result in an incorrect output from the network.
Backdoor Attack
User

Operational/ Model & Service

Internal Data

Client/ AI Pipeline
Collected Data

External Data Model (endpoint

deployment)
Backdoor Attack
Backdoor Attack
Confidential Customized for Lorem Ipsum LLC Version 1.0

THANK YOU!