Unit – 5

(Learning Notes)
SYLLABUS:
 MapReduce/Hadoop
 MapR M3
 Advanced SQL Tools

Hadoop Overview

Hortonworks founder predicted that by end of 2020, 75% of Fortune 2000

companies will be running 1000 node hadoop clusters in production. The
tiny toy elephant in the big data room has become the most popular big data
solution across the globe. However, implementation of Hadoop in production
is still accompanied by deployment and management challenges like
scalability, flexibility and cost effectiveness.

Many organizations that venture into enterprise adoption of Hadoop by

business users or by an analytics group within the company do not have
any knowledge on how a good hadoop architecture design should be and
how actually a hadoop cluster works in production. This lack of knowledge
leads to design of a hadoop cluster that is more complex than is necessary
for a particular big data application making it a pricey implementation.
Apache Hadoop was developed with the purpose of having a low–cost,
redundant data store that would allow organizations to leverage big data
analytics at economical cost and maximize profitability of the business.

A good hadoop architectural design requires various design considerations

in terms of computing power, networking and storage. This blog post gives
an in-depth explanation of the Hadoop architecture and the factors to be
considered when designing and building a Hadoop cluster for production
success.
Hadoop Architecture Overview
Apache Hadoop offers a scalable, flexible and reliable distributed
computing big data framework for a cluster of systems with storage capacity
and local computing power by leveraging commodity hardware. Hadoop
follows a Master Slave architecture for the transformation and analysis of
large datasets using Hadoop MapReduce paradigm. The 3 important hadoop
components that play a vital role in the Hadoop architecture are -
1. Hadoop Distributed File System (HDFS) – Patterned after the UNIX file
system
2. Hadoop MapReduce
3. Yet Another Resource Negotiator (YARN)

Hadoop follows a master slave architecture design for data storage and
distributed data processing using HDFS and MapReduce respectively. The
master node for data storage is hadoop HDFS is the NameNode and the
master node for parallel processing of data using Hadoop MapReduce is the
Job Tracker. The slave nodes in the hadoop architecture are the other
machines in the Hadoop cluster which store data and perform complex
computations. Every slave node has a Task Tracker daemon and a DataNode
that synchronizes the processes with the Job Tracker and NameNode
respectively. In Hadoop architectural implementation the master or slave
systems can be setup in the cloud or on-premise.
Role of Distributed Storage - HDFS in Hadoop Application
Architecture Implementation
A file on HDFS is split into multiple bocks and each is replicated
within the Hadoop cluster. A block on HDFS is a blob of data within the
underlying file system with a default size of 64MB.The size of a block can be
extended up to 256 MB based on the requirements.

Hadoop Distributed File System (HDFS) stores the application data and
file system metadata separately on dedicated servers. NameNode and
DataNode are the two critical components of the Hadoop HDFS architecture.
Application data is stored on servers referred to as DataNodes and file
system metadata is stored on servers referred to as NameNode. HDFS
replicates the file content on multiple DataNodes based on the replication
factor to ensure reliability of data. The NameNode and DataNode
communicate with each other using TCP based protocols. For the Hadoop
architecture to be performance efficient, HDFS must satisfy certain pre-
requisites –
1. All the hard drives should have a high throughput.
2. Good network speed to manage intermediate data transfer and block
replications.

NameNode
All the files and directories in the HDFS namespace are represented on the
NameNode by Inodes that contain various attributes like permissions,
modification timestamp, disk space quota, namespace quota and access
 times. NameNode maps the entire file system structure into memory. Two
files fsimage and edits are used for persistence during restarts.

 Fsimage file contains the Inodes and the list of blocks which define the
metadata.It has a complete snapshot of the file systems metadata at any
given point of time.

 The edits file contains any modifications that have been performed on the
content of the fsimage file.Incremental changes like renaming or appending
data to the file are stored in the edit log to ensure durability instead of
creating a new fsimage snapshot everytime the namespace is being altered.

DataNode
DataNode manages the state of an HDFS node and interacts with the blocks.
A DataNode can perform CPU intensive jobs like semantic and language
analysis, statistics and machine learning tasks, and I/O intensive jobs like
clustering, data import, data export, search, decompression, and indexing. A
DataNode needs lot of I/O for data processing and transfer.

On startup every DataNode connects to the NameNode and performs a

handshake to verify the namespace ID and the software version of the
DataNode. If either of them does not match then the DataNode shuts down
automatically. A DataNode verifies the block replicas in its ownership by
sending a block report to the NameNode. As soon as the DataNode registers,
the first block report is sent. DataNode sends heartbeat to the NameNode
every 3 seconds to confirm that the DataNode is operating and the block
replicas it hosts are available.

How does the Hadoop MapReduce architecture work?

The execution of a MapReduce job begins when the client submits the
job configuration to the Job Tracker that specifies the map, combine and
reduce functions along with the location for input and output data. On
receiving the job configuration, the job tracker identifies the number of
splits based on the input path and select Task Trackers based on their
network vicinity to the data sources. Job Tracker sends a request to the
selected Task Trackers.

The processing of the Map phase begins where the Task Tracker
extracts the input data from the splits. Map function is invoked for each
record parsed by the “InputFormat” which produces key-value pairs in the
memory buffer. The memory buffer is then sorted to different reducer nodes
by invoking the combine function. On completion of the map task, Task
Tracker notifies the Job Tracker. When all Task Trackers are done, the Job
Tracker notifies the selected Task Trackers to begin the reduce phase. Task
Tracker reads the region files and sorts the key-value pairs for each key. The
reduce function is then invoked which collects the aggregated values into
the output file.

MAPR M3 Architecture
MapR Database is an enterprise-grade, high performance, NoSQL (“Not Only
SQL”) database management system. You can use it to add realtime,
operational analytics capabilities to big data applications. As a multi-model
NoSQL database, it supports both JSON document models and wide column
data models.

Why use MapR Database?

 Integrated analytics with SQL: MapR Database's integration with
Drill for MapR provides a low latency, distributed, SQL-like query
engine for large-scale datasets, including structured and semi-
structured, nested data.

 Operational analytics: MapR Database can run in the same cluster

as Apache™ Hadoop® and Apache Spark, letting you immediately
analyze or process live, interactive data. This also enables you to
eliminate data silos to speed the data-to-action cycle, providing a more
efficient data architecture.

 Global distribution of applications: Application access to MapR

Database tables is distributable on a global scale.

 Flexible data model: You can use MapR Database as both a

document database and a wide-column database. As a document
database, MapR Database stores JSON documents in JSON tables. As
a wide-column database, it stores binary files in binary tables.
How is MapR Database Related to MapR
Filesystem?
MapR Database implements tables within the framework of the MapR file
system MapR Database creates tables (both binary and JSON tables) in
logical units called volumes.

MapR Database's architecture has the

following advantages:
 It reduces process overhead because it has no extra layers to pass
through when performing operations on data.

 MapR Database, like several other NoSQL databases, is a log-based

database. MapR Database runs inside of the MapR file system
process, which enables it to read from and write to disks directly. In
contrast, other NoSQL databases must communicate with a separate
process to performs disk reads and writes. The approach taken
by MapR Databaseeliminates extra process hops, duplicate caching,
and needless abstractions, with the consequence of optimizing I/O
operations on your data.

 It minimizes compaction delays because it avoids I/O storms when it

merges logged operations with structures on disk.
As a log-based database, MapR Database must write logged
operations to disk. MapR Database stores table regions (also
called tablets) and smaller structures within them partially as b-trees.
Together with write-ahead logs (WAL), these b-trees comprise log-
structured-merge trees. Write-ahead logs for the smaller structures
within regions are periodically restructured by rolling merge
operations on the b-trees. Because MapR Database performs these
merges at small scales, applications running against MapR
Database see no significant effects on latency while the merges are
taking place.
SQL Vs No-SQL