Getting Started With Apache Hadoop Cheatsheet

1. Introduction

Getting started with Apache Hadoop is a powerful ecosystem for handling big data. It allows you to store, process, and analyze vast amounts of data across distributed clusters of computers. Hadoop is based on the MapReduce programming model, which enables parallel processing of data. This section will cover the key components of Hadoop, its architecture, and how it works.

2. Installing Apache Hadoop

In this section, we’ll guide you through the installation process for Apache Hadoop. We’ll cover both single-node and multi-node cluster setups to suit your development and testing needs.

2.1 Single-Node Installation

To get started quickly, you can set up Hadoop in a single-node configuration on your local machine. Follow these steps:

Download Hadoop: Visit the Apache Hadoop website (https://hadoop.apache.org/) and download the latest stable release.

Extract the tarball: After downloading, extract the tarball to your preferred installation directory.

Set up environmental variables: Configure the HADOOP_HOME and add the Hadoop binary path to the PATH variable.

Configure Hadoop: Adjust the configuration files (core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml) to suit your setup.

Example code for setting environmental variables (Linux):

# Set HADOOP_HOME
export HADOOP_HOME=/path/to/hadoop

# Add Hadoop binary path to PATH
export PATH=$PATH:$HADOOP_HOME/bin

2.2 Multi-Node Installation

For production or more realistic testing scenarios, you’ll need to set up a multi-node Hadoop cluster. Here’s a high-level overview of the steps involved:

Prepare the machines: Set up multiple machines (physical or virtual) with the same version of Hadoop installed on each of them.

Configure SSH: Ensure passwordless SSH login between all the machines in the cluster.

Adjust the configuration: Modify the Hadoop configuration files to reflect the cluster setup, including specifying the NameNode and DataNode details.

3. Hadoop Distributed File System (HDFS)

HDFS is the distributed file system used by Hadoop to store large datasets across multiple nodes. It provides fault tolerance and high availability by replicating data blocks across different nodes in the cluster. This section will cover the basics of HDFS and how to interact with it.

3.1 HDFS Architecture

HDFS follows a master-slave architecture with two main components: the NameNode and the DataNodes.

3.1.1 NameNode

The NameNode is a critical component in the Hadoop Distributed File System (HDFS) architecture. It serves as the master node and plays a crucial role in managing the file system namespace and metadata. Let’s explore the significance of the NameNode and its responsibilities in more detail.

NameNode Responsibilities

Metadata Management: The NameNode maintains crucial metadata about the HDFS, including information about files and directories. It keeps track of the data block locations, replication factor, and other essential details required for efficient data storage and retrieval.

Namespace Management: HDFS follows a hierarchical directory structure, similar to a traditional file system. The NameNode manages this namespace, ensuring that each file and directory is correctly represented and organized.

Data Block Mapping: When a file is stored in HDFS, it is divided into fixed-size data blocks. The NameNode maintains the mapping of these data blocks to the corresponding DataNodes where the actual data is stored.

Heartbeat and Health Monitoring: The NameNode receives periodic heartbeat signals from DataNodes, which indicates their health and availability. If a DataNode fails to send a heartbeat, the NameNode marks it as unavailable and replicates its data to other healthy nodes to maintain data redundancy and fault tolerance.

Replication Management: The NameNode ensures that the configured replication factor for each file is maintained across the cluster. It monitors the number of replicas for each data block and triggers replication of blocks if necessary.

High Availability and Secondary NameNode

As the NameNode is a critical component, its failure could result in the unavailability of the entire HDFS. To address this concern, Hadoop introduced the concept of High Availability (HA) with Hadoop 2.x versions.

In an HA setup, there are two NameNodes: the Active NameNode and the Standby NameNode. The Active NameNode handles all client requests and metadata operations, while the Standby NameNode remains in sync with the Active NameNode. If the Active NameNode fails, the Standby NameNode takes over as the new Active NameNode, ensuring seamless HDFS availability.

Additionally, the Secondary NameNode is a misnomer and should not be confused with the Standby NameNode. The Secondary NameNode is not a failover mechanism but assists the primary NameNode in periodic checkpoints to optimize its performance. The Secondary NameNode periodically merges the edit logs with the fsimage (file system image) and creates a new, updated fsimage, reducing the startup time of the primary NameNode.

NameNode Federation

Starting from Hadoop 2.x, NameNode Federation allows multiple independent HDFS namespaces to be hosted on a single Hadoop cluster. Each namespace is served by a separate Active NameNode, providing better isolation and resource utilization in a multi-tenant environment.

NameNode Hardware Considerations

The NameNode’s role in HDFS is resource-intensive, as it manages metadata and handles a large number of small files. When setting up a Hadoop cluster, it’s essential to consider the following factors for the NameNode hardware:

By optimizing the NameNode hardware, you can ensure smooth HDFS operations and reliable data management in your Hadoop cluster.

3.1.2 DataNodes

DataNodes are integral components in the Hadoop Distributed File System (HDFS) architecture. They serve as the worker nodes responsible for storing and managing the actual data blocks that make up the files in HDFS. Let’s explore the role of DataNodes and their responsibilities in more detail.

DataNode Responsibilities

Data Storage: DataNodes are responsible for storing the actual data blocks of files. When a file is uploaded to HDFS, it is split into fixed-size blocks, and each block is stored on one or more DataNodes. The DataNodes efficiently manage the data blocks and ensure their availability.

Data Block Replication: HDFS replicates data blocks to provide fault tolerance and data redundancy. The DataNodes are responsible for creating and maintaining replicas of data blocks as directed by the NameNode. By default, each data block is replicated three times across different DataNodes in the cluster.

Heartbeat and Block Reports: DataNodes regularly send heartbeat signals to the NameNode to indicate their health and availability. Additionally, they provide block reports, informing the NameNode about the list of blocks they are storing. The NameNode uses this information to track the availability of data blocks and manage their replication.

Data Block Operations: DataNodes perform read and write operations on the data blocks they store. When a client wants to read data from a file, the NameNode provides the locations of the relevant data blocks, and the client can directly retrieve the data from the corresponding DataNodes. Similarly, when a client wants to write data to a file, the data is written to multiple DataNodes based on the replication factor.

DataNode Health and Decommissioning

DataNodes are crucial for the availability and reliability of HDFS. To ensure the overall health of the Hadoop cluster, the following factors related to DataNodes are critical.

Heartbeat and Health Monitoring: The NameNode expects periodic heartbeat signals from DataNodes. If a DataNode fails to send a heartbeat within a specific time frame, the NameNode marks it as unavailable and starts the process of replicating its data blocks to other healthy nodes. This mechanism helps in quickly detecting and recovering from DataNode failures.

Decommissioning: When a DataNode needs to be taken out of service for maintenance or other reasons, it goes through a decommissioning process. During decommissioning, the DataNode informs the NameNode about its intent to leave the cluster gracefully. The NameNode then starts replicating its data blocks to other nodes to maintain the desired replication factor. Once the replication is complete, the DataNode can be safely removed from the cluster.

DataNode Hardware Considerations

DataNodes are responsible for handling a large amount of data and performing read and write operations on data blocks. When setting up a Hadoop cluster, consider the following factors for DataNode hardware:

By optimizing the hardware for DataNodes, you can ensure smooth data operations, fault tolerance, and high availability within your Hadoop cluster.

3.2 Interacting with HDFS

You can interact with HDFS using either the command-line interface (CLI) or the Hadoop Java API. Here are some common HDFS operations:

Uploading files to HDFS:

hadoop fs -put /local/path/to/file /hdfs/destination/path

Downloading files from HDFS:

hadoop fs -get /hdfs/path/to/file /local/destination/path

Listing files in a directory:

hadoop fs -ls /hdfs/path/to/directory

Creating a new directory in HDFS:

hadoop fs -mkdir /hdfs/new/directory

4. MapReduce

MapReduce is the core programming model of Hadoop, designed to process and analyze vast datasets in parallel across the Hadoop cluster. It breaks down the processing into two phases: the Map phase and the Reduce phase. Let’s dive into the details of MapReduce.

4.1 MapReduce Workflow

The MapReduce workflow consists of three steps: Input, Map, and Reduce.

Input: The input data is divided into fixed-size splits, and each split is assigned to a mapper for processing.

Map: The mapper processes the input splits and produces key-value pairs as intermediate outputs.

Shuffle and Sort: The intermediate key-value pairs are shuffled and sorted based on their keys, grouping them for the reduce phase.

Reduce: The reducer processes the sorted intermediate data and produces the final output.

4.2 Writing a MapReduce Job

To write a MapReduce job, you’ll need to create two main classes: Mapper and Reducer. The following is a simple example of counting word occurrences in a text file.

Mapper class:

import java.io.IOException;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;

public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
  private final static IntWritable one = new IntWritable(1);
  private Text word = new Text();

  @Override
  public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
    String[] words = value.toString().split("\\s+");
    for (String w : words) {
      word.set(w);
      context.write(word, one);
    }
  }
}

Reducer class:

import java.io.IOException;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;

public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
  @Override
  public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
    int sum = 0;
    for (IntWritable val : values) {
      sum += val.get();
    }
    context.write(key, new IntWritable(sum));
  }
}

Main class:

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {
  public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "word count");
    job.setJarByClass(WordCount.class);
    job.setMapperClass(WordCountMapper.class);
    job.setReducerClass(WordCountReducer.class);
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));


    System.exit(job.waitForCompletion(true) ? 0 : 1);
  }
}

5. Apache Hadoop Ecosystem

Apache Hadoop has a rich ecosystem of related projects that extend its capabilities. In this section, we’ll explore some of the most popular components of the Hadoop ecosystem.

5.1 Apache Hive

Using Apache Hive involves several steps, from creating tables to querying and analyzing data. Let’s walk through a basic workflow for using Hive.

5.1.1 Launching Hive and Creating Tables

Start Hive CLI (Command Line Interface) or use HiveServer2 for a JDBC/ODBC connection.

Create a database (if it doesn’t exist) to organize your tables:

CREATE DATABASE mydatabase;

Switch to the newly created database:

USE mydatabase;

Define and create a table in Hive, specifying the schema and the storage format. For example, let’s create a table to store employee information:

CREATE TABLE employees (
  emp_id INT,
  emp_name STRING,
  emp_salary DOUBLE
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',';

5.1.2 Loading Data into Hive Tables

Upload data files to HDFS or make sure the data is available in a compatible storage format (e.g., CSV, JSON) accessible by Hive.

Load the data into the Hive table using the LOAD DATA command. For example, if the data is in a CSV file located in HDFS:

LOAD DATA INPATH '/path/to/employees.csv' INTO TABLE employees;

5.1.3 Querying Data with Hive

Now that the data is loaded into the Hive table, you can perform SQL-like queries on it using Hive Query Language (HQL). Here are some example queries:

Retrieve all employee records:

SELECT * FROM employees;

Calculate the average salary of employees:

SELECT AVG(emp_salary) AS avg_salary FROM employees;

Filter employees earning more than $50,000:

SELECT * FROM employees WHERE emp_salary > 50000;

5.1.4 Creating Views in Hive

Hive allows you to create views, which are virtual tables representing the results of queries. Views can simplify complex queries and provide a more user-friendly interface. Here’s how you can create a view:

CREATE VIEW high_salary_employees AS
SELECT * FROM employees WHERE emp_salary > 75000;

5.1.5 Using User-Defined Functions (UDFs)

Hive allows you to create custom User-Defined Functions (UDFs) in Java, Python, or other supported languages to perform complex computations or data transformations. After creating a UDF, you can use it in your HQL queries. For example, let’s create a simple UDF to convert employee salaries from USD to EUR:

package com.example.hive.udf;

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

public class USDtoEUR extends UDF {
  public Text evaluate(double usd) {
    double eur = usd * 0.85; // Conversion rate (as an example)
    return new Text(String.valueOf(eur));
  }
}

Compile the UDF and add the JAR to the Hive session:

ADD JAR /path/to/usd_to_eur_udf.jar;

Then, use the UDF in a query:

SELECT emp_id, emp_name, emp_salary, USDtoEUR(emp_salary) AS emp_salary_eur FROM employees;

5.1.6 Storing Query Results

You can store the results of Hive queries into new tables or external files. For example, let’s create a new table to store high-earning employees:

CREATE TABLE high_earning_employees AS
SELECT * FROM employees WHERE emp_salary > 75000;

5.1.7 Exiting Hive

Once you have completed your Hive operations, you can exit the Hive CLI or close your JDBC/ODBC connection.

This is just a basic overview of using Hive. Hive is a powerful tool with many advanced features, optimization techniques, and integration options with other components of the Hadoop ecosystem. As you explore and gain more experience with Hive, you’ll discover its full potential for big data analysis and processing tasks.

5.2 Apache Pig

Apache Pig is a high-level data flow language and execution framework built on top of Apache Hadoop. It provides a simple and expressive scripting language called Pig Latin for data manipulation and analysis. Pig abstracts the complexity of writing low-level Java MapReduce code and enables users to process large datasets with ease. Pig is particularly useful for users who are not familiar with Java or MapReduce but still need to perform data processing tasks on Hadoop.

Want to master Apache Hadoop

Subscribe to our newsletter and download the Getting Started with Apache Hadoop Cheatsheet right now!

In order to help you master Apache Hadoop, we have compiled a kick-ass all-in-one cheatsheet with all the popular and most used cases! Besides studying them online, you may also download the Cheatsheet in PDF format!!

Download Now

Thank you!

We will contact you soon.

5.2.1 Pig Latin

Pig Latin is the scripting language used in Apache Pig. It consists of a series of data flow operations, where each operation takes input data, performs a transformation, and generates output data. Pig Latin scripts are translated into a series of MapReduce jobs by the Pig execution engine.

Pig Latin scripts typically follow the following structure:

-- Load data from a data source (e.g., HDFS)
data = LOAD '/path/to/data' USING PigStorage(',') AS (col1:datatype, col2:datatype, ...);

-- Data transformation and processing
transformed_data = FOREACH data GENERATE col1, col2, ...;

-- Filtering and grouping
filtered_data = FILTER transformed_data BY condition;
grouped_data = GROUP filtered_data BY group_column;

-- Aggregation and calculations
aggregated_data = FOREACH grouped_data GENERATE group_column, SUM(filtered_data.col1) AS total;

-- Storing the results
STORE aggregated_data INTO '/path/to/output' USING PigStorage(',');

5.2.2 Pig Execution Modes

Pig supports two execution modes.

Local Mode: In Local Mode, Pig runs on a single machine and uses the local file system for input and output. It is suitable for testing and debugging small datasets without the need for a Hadoop cluster.

MapReduce Mode: In MapReduce Mode, Pig runs on a Hadoop cluster and generates MapReduce jobs for data processing. It leverages the full power of Hadoop’s distributed computing capabilities to process large datasets.

5.2.3 Pig Features

5.2.4 Using Pig

To use Apache Pig, follow these general steps:

Install Apache Pig on your Hadoop cluster or a standalone machine.

Write Pig Latin scripts to load, transform, and process your data. Save the scripts in .pig files.

Run Pig in either Local Mode or MapReduce Mode, depending on your data size and requirements.

Here’s an example of a simple Pig Latin script that loads data, filters records, and stores the results:

-- Load data from HDFS
data = LOAD '/path/to/input' USING PigStorage(',') AS (name:chararray, age:int, city:chararray);

-- Filter records where age is greater than 25
filtered_data = FILTER data BY age > 25;

-- Store the filtered results to HDFS
STORE filtered_data INTO '/path/to/output' USING PigStorage(',');

As you become more familiar with Pig, you can explore its advanced features, including UDFs, joins, groupings, and more complex data processing operations. Apache Pig is a valuable tool in the Hadoop ecosystem, enabling users to perform data processing tasks efficiently without the need for extensive programming knowledge.

5.3 Apache HBase

Apache HBase is a distributed, scalable, and NoSQL database built on top of Apache Hadoop. It provides real-time read and write access to large amounts of structured data. HBase is designed to handle massive amounts of data and is well-suited for use cases that require random access to data, such as real-time analytics, online transaction processing (OLTP), and serving as a data store for web applications.

5.3.1 HBase Features

5.3.2 HBase Architecture

HBase follows a master-slave architecture with the following key components:

5.3.3 Using HBase

To use Apache HBase, follow these general steps:

Install Apache HBase on your Hadoop cluster or a standalone machine.

Start the HBase Master and RegionServers.

Create HBase tables and specify the column families.

Use the HBase API or HBase shell to perform read and write operations on HBase tables.

Here’s an example of using the HBase shell to create a table and insert data:

$ hbase shell

hbase(main):001:0> create 'my_table', 'cf1', 'cf2'
hbase(main):002:0> put 'my_table', 'row1', 'cf1:col1', 'value1'
hbase(main):003:0> put 'my_table', 'row1', 'cf2:col2', 'value2'
hbase(main):004:0> scan 'my_table'

This example creates a table named my_table with two column families (cf1 and cf2), inserts data into rows row1, and scans the table to retrieve the inserted data.

Apache HBase is an excellent choice for storing and accessing massive amounts of structured data with low-latency requirements. Its integration with the Hadoop ecosystem makes it a powerful tool for real-time data processing and analytics.

5.4 Apache Spark

Apache Spark is an open-source distributed data processing framework designed for speed, ease of use, and sophisticated analytics. It provides an in-memory computing engine that enables fast data processing and iterative algorithms, making it well-suited for big data analytics and machine learning applications. Spark supports various data sources, including Hadoop Distributed File System (HDFS), Apache HBase, Apache Hive, and more.

5.4.1 Spark Features

In-Memory Computing: Spark keeps intermediate data in memory, reducing the need to read and write to disk and significantly speeding up data processing.

Resilient Distributed Dataset (RDD): Spark’s fundamental data structure, RDD, allows for distributed data processing and fault tolerance. RDDs are immutable and can be regenerated in case of failures.

Data Transformation and Actions: Spark provides a wide range of transformations (e.g., map, filter, reduce) and actions (e.g., count, collect, save) for processing and analyzing data.

Spark SQL: Spark SQL enables SQL-like querying on structured data and seamless integration with data sources like Hive and JDBC.

MLlib: Spark’s machine learning library, MLlib, offers a rich set of algorithms and utilities for building and evaluating machine learning models.

GraphX: GraphX is Spark’s library for graph processing, enabling graph analytics and computations on large-scale graphs.

Spark Streaming: Spark Streaming allows real-time processing of data streams, making Spark suitable for real-time analytics.

5.4.2 Spark Architecture

Spark follows a master-slave architecture with the following key components:

5.4.3 Using Apache Spark

To use Apache Spark, follow these general steps:

Install Apache Spark on your Hadoop cluster or a standalone machine.

Create a SparkContext, which is the entry point to Spark functionalities.

Load data from various data sources into RDDs or DataFrames (Spark SQL).

Perform transformations and actions on the RDDs or DataFrames to process and analyze the data.

Use Spark MLlib for machine learning tasks if needed.

Save the results or write the data back to external data sources if required.

Here’s an example of using Spark in Python to count the occurrences of each word in a text file:

from pyspark import SparkContext

# Create a SparkContext
sc = SparkContext("local", "Word Count")

# Load data from a text file into an RDD
text_file = sc.textFile("path/to/text_file.txt")

# Split the lines into words and count the occurrences of each word
word_counts = text_file.flatMap(lambda line: line.split(" ")).map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)

# Print the word counts
for word, count in word_counts.collect():
    print(f"{word}: {count}")

# Stop the SparkContext
sc.stop()

Apache Spark’s performance, ease of use, and broad range of functionalities have made it a popular choice for big data processing, analytics, and machine learning applications. Its ability to leverage in-memory computing and seamless integration with various data sources and machine learning libraries make it a versatile tool in the big data ecosystem.

5.5 Apache Sqoop

Apache Sqoop is an open-source tool designed for efficiently transferring data between Apache Hadoop and structured data stores, such as relational databases. Sqoop simplifies the process of importing data from relational databases into Hadoop’s distributed file system (HDFS) and exporting data from HDFS to relational databases. It supports various databases, including MySQL, Oracle, PostgreSQL, and more.

5.5.1 Sqoop Features

Data Import and Export: Sqoop allows users to import data from relational databases into HDFS and export data from HDFS back to relational databases.

Parallel Data Transfer: Sqoop uses multiple mappers in Hadoop to import and export data in parallel, achieving faster data transfer.

Full and Incremental Data Imports: Sqoop supports both full and incremental data imports. Incremental imports enable transferring only new or updated data since the last import.

Data Compression: Sqoop can compress data during import and decompress it during export, reducing storage requirements and speeding up data transfer.

Schema Inference: Sqoop can automatically infer the database schema during import, reducing the need for manual schema specification.

Integration with Hadoop Ecosystem: Sqoop integrates seamlessly with other Hadoop ecosystem components, such as Hive and HBase, enabling data integration and analysis.

5.5.2 Sqoop Architecture

Sqoop consists of the following key components:

5.5.3 Using Apache Sqoop

To use Apache Sqoop, follow these general steps:

Install Apache Sqoop on your Hadoop cluster or a standalone machine.

Configure the Sqoop Client by specifying the database connection details and other required parameters.

Use the Sqoop Client to import data from the relational database into HDFS or export data from HDFS to the relational database.

Here’s an example of using Sqoop to import data from a MySQL database into HDFS:

# Import data from MySQL to HDFS
sqoop import \
  --connect jdbc:mysql://mysql_server:3306/mydatabase \
  --username myuser \
  --password mypassword \
  --table mytable \
  --target-dir /user/hadoop/mydata

This example imports data from the mytable in the MySQL database into the HDFS directory /user/hadoop/mydata.

Apache Sqoop simplifies the process of transferring data between Hadoop and relational databases, making it a valuable tool for integrating big data with existing data stores and enabling seamless data analysis in Hadoop.

6. Additional Resources

Here are some additional resources to learn more about the topics mentioned:

Additionally, you can find many tutorials, blog posts, and online courses on platforms like Udemy, Coursera, and LinkedIn Learning that offer in-depth knowledge on these Apache projects. Happy learning!

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.