Big Data for IoT, Cloud, and AI

Big Data for IoT, Cloud, and AI

Big Data for IoT, Cloud, and AI

Anasooya Khanna

Big Data for IoT, Cloud, and AI

Anasooya Khanna

ISBN - 9789361520327

COPYRIGHT © 2025 by Educohack Press. All rights reserved.

This work is protected by copyright, and all rights are reserved by the Publisher. This includes, but is not limited to, the rights to translate, reprint, reproduce, broadcast, electronically store or retrieve, and adapt the work using any methodology, whether currently known or developed in the future.

The use of general descriptive names, registered names, trademarks, service marks, or similar designations in this publication does not imply that such terms are exempt from applicable protective laws and regulations or that they are available for unrestricted use.

The Publisher, authors, and editors have taken great care to ensure the accuracy and reliability of the information presented in this publication at the time of its release. However, no explicit or implied guarantees are provided regarding the accuracy, completeness, or suitability of the content for any particular purpose.

If you identify any errors or omissions, please notify us promptly at educohackpress@gmail.com & sales@educohackpress.com We deeply value your feedback and will take appropriate corrective actions.

The Publisher remains neutral concerning jurisdictional claims in published maps and institutional affiliations.

Published by Educohack Press, House No. 537, Delhi- 110042, INDIA

Email: educohackpress@gmail.com & sales@educohackpress.com

Cover design by Team EDUCOHACK

Preface

In a world full of human beings and an abundance of information Data is ubiquitous. Managing such vast data becomes a very hectic task with traditional methods. However, it is important to manage the data to target relevant and useful information for speeding and improving our work. Therefore, in current times there is a big demand for database engineers and Data Analysts.

This book will highlight popular advanced methods used for data managing, which successfully makes this hectic task very easy and understandable. The most popular data management techniques are Big Data, Data Mining and Machine Learning. And this book will enlighten you with all these techniques which you can use to enhance your knowledge.

Big Data has been used in the industry for extracting relevant information from large data to gain profit. Companies use data mining with a strong consumer focus — retail, financial, communication, and marketing organizations, to drill down into their transactional data and determine price, customer preferences, and product positioning, impacting sales, customer satisfaction, and corporate profits. Whereas, Machine learning mainly focuses on improving and including various features in a website or application like google maps, etc.

This book will contain all the details regarding these techniques and fulfill all your questions regarding the above topics. This book will also give you a chance to test your knowledge as there are questions related to each topic. As a bonus, you will find reference links to dive more into the topic. I hope you will be satisfied and will be able to gain useful knowledge to shape your career.

Happy Reading!!

Content

Part-I

Overview Of Big Data

01. Overview of Big Data

1.1 History of Big Data 3

1.2 Examples of Big Data 4

1.3 Impacts of Big data 5

1.4 Future Expectations 6

1.5 Career Scope 7

1.6 Tools Used 10

1.7 Summary 14

1.9 Exercise 15

02. Types of Big Data

2.1 Structured Data 19

2.2 Unstructured Data 19

2.3 Semi-Structured Data 20

2.4 Difference between types of big data 20

2.5 Summary 21

2.6 Exercise 21

03. Characteristics of Big Data 24

3.1 Volume 25

3.3 Veracity 26

3.4 Value 26

3.5 Velocity 27

3.6 Summary 27

3.8 Exercise 28

04. Working of Big Data

4.1 Integration 30

4.2 Management 31

4.3 Analysis 32

4.4 Big Data practices 33

4.5 Summary 35

4.6 Exercise 35

05. Advantages and Disadvantages of Big Data

5.1 Benefits 38

5.2 Advantages of Big Data 39

5.3 Disadvantages of Big Data 41

5.4 Summary 42

5.5 Exercise 42

06. Real-Time Applications

6.1 Fields where big data is being used 45

6.2 Companies that use big data 54

6.3 Companies that provide big data 54

6.4 Summary 55

6.5 Exercise 55

07. Case Studies

7.1 What are case studies? 58

7.2 Case Study 1 – Walmart 58

7.3 Case Study 2 – Uber 59

7.4 Case Study 3 – Netflix 59

7.5 Case Study 4 – eBay 59

7.6 Case Study 5 – Procter & Gamble 60

7.7 Case Study 6 – LinkedIn 60

7.8 Summary 60

Part-II

DATA MINING

08. Overview of Data Mining

8.1 What do you mean by Data Mining? 65

8.2 History of Data Mining 66

8.3 Tasks Related 67

8.3. Summarization 70

8.4 Career Scope 71

8.5 Future of Data Mining 73

8.6 Data Mining V/S Big Data 74

8.7 Summary 74

8.8 Exercise 75

09. Stages Of Data Mining Process

9.1 Data Purification 78

9.2 Data Integration 78

9.3 Data Selection 79

9.4 Data Transformation 80

9.5 Pattern Evaluation 81

9.6 Knowledge Representation 82

9.7 Summary 82

9.8 Exercise 83

010. Data Mining Techniques

10.1 Tracking patterns 86

10.2 Classification 86

10.3 Clustering 86

10.4 Outlier detection 87

10.5 Association 87

10.6 Regression 88

10.7 Prediction 89

10.8 Sequential patterns 89

10.9 Decision trees 89

10.10 Statistical techniques 90

10.11 Visualization 91

10.12 Neural networks 91

10.13 Data warehousing 92

10.14 Long-term memory processing 93

10.15 Machine learning and AI 93

10.16 Summary 93

10.17 Exercises 94

11. Tools Used In Data Mining

11.1 MonkeyLearn 96

11.2 RapidMiner 97

11.3 Oracle Data Mining 98

11.4 IBM SPSS Modeler 99

11.5 Weka 100

11.6 KNIME 100

11.7 H2O 101

11.8 Orange 102

11.9 Apache Mahout 102

11.10 SAS Enterprise Mining 103

11.11 Rattle 104

11.12 Python 105

11.13 Summary 106

11.14 Exercise 107

12. Data Mining Algorithms

12.1 C4.5 109

12.2 k-means 110

12.3 Support vector machines 112

12.4 Apriori 113

12.5 EM 115

12.6 Page Rank 117

12.7 AdaBoost 117

12.8 kNN 119

12.9 Naive Bayes 120

12.10 CART 121

12.13 Summary 122

12.14 Exercise 122

13. Applications and Drawbacks of Data Mining

13.1 Future Healthcare 126

13.2 Market Based Analysis 126

13.3 Manufacturing Engineering 127

13.4 Education 128

13.5 CRM 129

13.6 Fraud Detection 130

13.7 Intrusion Detection 131

13.8 Lie Detection 132

13.9 Customer Segmentation 133

13.10 Financial Banking 134

13.11 Corporate Surveillance 135

13.12 Research Analysis 136

13.13 Criminal Investigation 136

13.14 BioInformatics 137

13.15 Companies extensively using Data Mining 138

13.16 Drawbacks of Data mining 139

13.17 Summary 140

13.18 Exercise 141

Part-III

MACHINE LEARNING

14. Introduction to Machine Learning

14.1 What is ML? 148

14.2 History of ML 148

14.3 How is it different from AI? 149

14.4 Examples of Machine Learning 150

14.5 Demand Of Machine Learning 158

14.6 Summary 159

14.7 Exercise 159

15. Types of Machine Learning

15.1 Supervised Learning 162

15.2 Examples of Supervised Learning 163

15.3 Unsupervised Learning 164

15.4 Examples of Unsupervised Learning 165

15.5 Reinforcement Learning 165

15.6 Examples of Reinforcement Learning 166

15.7 Summary 167

15.8 Exercise 167

16. Models in Machine Learning

16.1 Classification models 170

16.2 Regression models 177

16.3 Clustering models 182

16.3.4 EM with GMM 185

16.4 Association models 187

16.5 Reinforcement learning models 191

16.6 Difference between ML models 198

16.7 Summary 199

16.8 Exercise 200

17. Emerging Trends in ML

17.1 Intersection Of ML and IoT 204

17.2 Automated Machine Learning 205

17.3 Machine Learning in Cyber Security 206

17.4 Rise of AI Ethics 207

17.5 AI Engineering 207

17.6 AI-driven Biometric Security Solution 208

17.7 AI Analysis for the business forecast 208

17.8 Companies dependent on ML 209

17.9 Summary 210

17.10 Exercise 210

18. Model building in ML

18.1 Defining the problem 213

18.2 Collecting Data 214

18.3 Measure of success 215

18.4 Evaluation Protocol 215

18.5 Preparing the Data 218

18.6 Benchmark model 225

18.7 Tunning the hyperparameters 226

18.8 Summary 231

18.9 Exercise 232

Glossary 235

Index244

Part 1

Part 1

Overview of Big Data

Chapter 1. Exploring Big Data

Abstract

This chapter throws light on Big Data and its importance in our lives. Also, it will enlighten us about how big data can bring improvements and profit if it is managed properly. Here we will see the whole history and how big data has become an essential requirement for today’s generation. This will explain the future needs and increasing demands of big data. The career scope is also covered in this chapter. It will give you an overview and clear the importance of big data with its excellent uses. So keep reading to dive into the world of big data.

1.1 History of Big Data

The term big data was first used in 2005 by O’Reilly Media. Big data is mainly used for tracking down one’s business and making improvements. This concept is not new; it was used 7000 years ago in Mesopotamia when accounting was introduced there. The first major project was carried out in 1937, ordered by Franklin D. Roosevelt’s administration in the USA. The government had to keep track of contributions from 26 million Americans, and more than 2 million employers as the Social Security Act became law in 1937. Hereafter many such projects were conducted. The first data processing machine was developed in 1943 by the British to decipher Nazi codes in World War II.

In 2005, for the first time, the term Big Data was coined by Roger Mougalas from O’Reilly Media. In the same year, Hadoop was created by yahoo. Its goal was to index the entire World Wide Web. Nowadays, Hadoop is used by a lot of organizations to crunch through huge data. Afterward, increasing social media boosted Big Data, and it became the essential demand of every existing and emerging company.

Fig. 1.1 Evolution of Big Data

1.2 Examples of Big Data

Nowadays, Big Data is everywhere; most companies use it for gaining profit and making improvements in their business. Here are some of the examples:

Fig. 1.2 Applications of Big Data

1.2.1 Marketing

Big Data has become an essential tool for marketing. Most companies rely on big data to improve their sales. It saves marketers time as big data allows them to advertise their ads to a targeted audience. It has entered into every sector, resulting in advanced marketing and gaining maximum profits quickly. This can be seen in many ways, like How Amazon shows the correct ad related to your interests? The suggestions of accurate products while shopping, etc., can be achieved using big data.

1.2.2 Transportation

In this modern world, everyone has a smartphone, but what makes it smart is its features. Almost everyone, while traveling, uses maps to see the shortest route or traffic areas, etc., for their convenience. Transportation relies on big data for achieving better and safe transport from one place to another. It allows them to navigate and detect people, cars, traffic expected, etc. Tracking down the parcels is also one of the greatest examples.

1.2.3 Business Insights

Most companies collect large amounts of data and analytics to gain customers’ requirements and fulfill them. Using big data enables them to understand their online audience, their expectations and predict future buying. It leads to an increase in customer satisfaction and helps to build trust among them.

1.3 Impacts of Big data

The impacts of big data are in every sector. They use it for the improvement and benefit of gaining people’s trust and attention. As an audience, we find it quite appealing when companies or organizations bring us news or products as per our needs which helps the company gain a reputation in the market. Here are some impacts of Big data :

� Manufacturing

� Logistics, Media, and Entertainment

� Oil and Gas

� Social Media, etc.

� We will see detailed information further in upcoming chapters.

1.4 Future Expectations

Fig. 1.3 Future of Big Data

1.4.1 Data volumes will continue to increase and migrate to the cloud

The majority of Big Data experts agree that a very large amount of data will be generated in the upcoming years. Companies will continue to analyze these tons of data. They require large data sets and infinite data stacks to avoid this problem; the whole data is stored in the cloud, such as AWS. This creates a challenge for big data processing though it has become easier through open-source ecosystems such as Hadoop and No SQL.

1.4.2 Machine learning will continue to change the landscape

Apart from Big Data, one of the future technologies includes Machine Learning, which will greatly impact our future changing lifestyles. Combining big data with ML will automate big data analysis, where computers will analyze the data by themselves.

1.4.3 Data scientists and CDOs will be in high demand

The positions of Data scientists and CDOs, i.e., Chief Data Officers, will be in high demand. In the current times, the gap between the need and data professionals is high, So in the future, there will be a great demand for data professionals as with increasing data, we need more people to manage it. No doubt that Data Analysts, data engineers are one of the fastest-growing jobs.

1.4.4 Fast data and actionable data will come to the forefront

Big Data future is also related to Fast data and Actionable data. Big Data relies on Hadoop and No SQL database to analyze information, whereas Fast data allows processing in real-time streams. As a result, data can be analyzed in just one millisecond. One of the research predicted that nearly 30% of data would be real-time by 2025.

1.5 Career Scope

Observing the benefits of big data demand of data professionals has been in high demand. Depending on your positions and skills, big data jobs are lucrative. It also secures a nice living for you and your family. The pay range lies between the range of $ 50,000 - $165,000. The topmost jobs in the field of Big Data are engineers, managers, and developers. Also, some highest-paying big data positions include analysts, scientists, statisticians, and specialists. You can easily find a job in big data as there are plenty of jobs available. Most of Big data positions rely on solid programming background. You should at least know the basics of C, Python, Java, and SQL. To make your profile stronger, you may require certifications in Hadoop, Apache Spark, and Machine learning.