SMA PROCESS/STEPS
OVERVIEW
STEPS:
1. Authorization
2. Data collection
3. Data pre-processing
4. Modelling and analysis
5. Results
1. AUTHORIZATION:
• In authorization we make use of industry standard framework called “OAuth”.
• The OAuth stands for “Open Authorization”.
The process involves three actors:
1. User
2. Consumer(Application)
3. Resource provider(Social media platform)
• The user agrees with the consumer to grant access to the social media platform
• As the user doesn’t give their social media password directly to the
consumer.
• The consumer has initial exchange with the resource provider to generate a
token and a secret.
• The user is then redirected with the token to the resource provider, which will
ask to confirm authorizing the consumer to access the user’s data.
• Depending on the nature of social media platform, it will also ask to confirm
whether the consumer can perform any action on the user’s behalf, for
example: Post an update, share a link so on.
• The resource provider issues a valid token for the consumer.
• The token can then go back to the user confirming the access.
2. DATA COLLECTION:
Once we get the authorized access to social media platforms such as twitter,
Facebook etc then we will collect the data using tokens.
The data collection step is tied with authorization process as we can only
download data that we have been granted access to.
Initially the data will be collected in raw form(unstructured).
The data will downloaded in json format(JavaScript Object Notation.)
3. DATA PRE-PROCESSING:
The collected data will be in unstructured format.
The data may have some outliers, missing values, biased data.
By applying various data pre-processing techniques, we
will clean and process the and making it model
development ready.
DATA-PREPROCESSING INVOLVES:
• Missing value identification & treatment
• Outlier analysis
• Feature extraction
• Dimensionality reduction(Feature selection)
• Identifying the noisy data.
• Identifying duplicated data.
• Data transformation
• Data integration
4. MODELLING AND DATA ANALYSIS
The modelling and analysis can be done by the component called “ANALYTICS
ENGINE”.
The modelling and analysis can be encountered by text mining and graph mining.
Analytics engines employ various analytical techniques and algorithms to derive
insights from data.
This may involve descriptive analytics to summarize historical data patterns.
Diagnostic analytics to identify the root causes of problems.
Predictive analytics to forecast future trends.
Prescriptive analytics to recommend actions based on insights.
TEXT MINING:
Document classification: This is the task of assigning a document to one or more
categories
Document clustering: This is the task of grouping documents into subsets (called
clusters) that are coherent and distinct from one another (for example, by topic or
sub-topic).
Document summarization: This is the task of creating a shortened version of the
document in order to reduce the information overload to the user, while still
retaining the most important aspects described in the original source
Sentiment analysis: This is the task of identifying and categorizing sentiments and
opinions expressed in a text in order to understand the attitude towards a particular
product, topic, service, and so on
5. RESULTS
• Once the modelling and analysis is done then will get the results.
• The results can be communicated to data scientist in the form reports,
presentations etc.,
• In advance we can represent the results in the form of web applications,
dasboards etc.
• Here the results are evaluated to check whether it’s reaching the expected
objectives or not.
UNDERSTANDING PROCESS
Problem definition: Understanding the problem is crucial for choosing the right data sources and
the methods of analysis. It also helps to realize what kind of information and conclusions we can
infer from the data and what is impossible to derive.
Getting The Right Data: according to the defined problem. Many social media platforms allow
users to collect a lot of information in an automatized way via APIs (Application Programming
Interfaces). However, other platforms, such as forums or blogs, usually require a customized
programming approach
Cleaning: Involves tasks such as duplicates removal (retweets on Twitter), or spam detection to
remove irrelevant comments, language detection to perform linguistic analysis, or other statistical or
machine learning approaches that can help to produce a clean dataset.
Data Analysis: The choice of the analysis method depends on the objectives of the study and the
type of data. It may require statistical or machine learning approach, or a specific approach to time
series.
Data Visualization: Data and output can be presented in many different ways, starting from charts,
plots, and diagrams through more complex 2D charts, to multidimensional visualizations
