Friday, 23 February 2018

File Write in HDFS - Hadoop Framework Internal Steps

In this post we’ll see what all happens internally with in the Hadoop framework when a file is written in HDFS.

Writing file in HDFS

When client application wants to create a file in HDFS it calls create() method on DistributedFileSystem which in turn calls the create() method of the DFSClient. With in the Hadoop framework it is the DFSClient class which communicates with NameNode and DataNodes.

Since the metadata about the file is stored in NameNode so first communication is with NameNode to store file data in its namespace. NameNode performs some checks related to client permissions and file not already existing. If those checks are passed metadata about the file is stored.

For creating Output Stream FSDataOutputStream and DFSOutputStream classes are used. Client streams file data to FSDataOutputStream which wraps an instance of DFSOutputStream class. The streamed file data is cached internally by DFSOutputStream class and divided into packets of 64 KB. These packets are enqueued into a queue called dataQueue which is actually a Java LinkedList.

There is another class DataStreamer which connects to NameNode and retrieves a new blockid and block locations. Remember for the default replication factor of 3 NameNode will send a list of 3 DataNodes for storing each block of the file. These DataNodes form a pipeline to stream block data from one DataNode to another.

The DataStreamer thread picks up packets from the dataQueue and streams it to the first datanode in the pipeline which stores the packets and also forwards those packets to the second Datanode which stores them and forwards the packet to the third Datanode in the pipeline.

There is another queue in the DFSOutputStream class known as ackQueue. After sending the packet to the first datanode in the pipeline DataStreamer thread moves that packet from the dataQueue to the ackQueue. When that packet is written to all the DataNodes in the pipeline then only it is removed from ackQueue.

In case any of the DataNode in the pipeline fails then the pipeline is closed and the packets in the ackQueue that are not yet acknowledged are removed from the ackQueue and added in the front of dataQueue so that the DataStreamer thread can pick them up again.

Then a new pipeline is created excluding the failed DataNode and the packets are written to the remaining two DataNodes.

When each DataNode in the pipeline has completed writing the block locally, DataNode also notify the NameNode of their block storage. For the scenario as stated above where block is replicated to only two DataNodes, after getting the reports from DataNodes, NameNode will know that the replication factor of 3 is not maintained so it will ensure that the replica is created in another DataNode.

HDFS data flow for file write in HDFS

That's all for this topic File Write in HDFS - Hadoop Framework Internal Steps. If you have any doubt or any suggestions to make please drop a comment. Thanks!


Related Topics

  1. File Read in HDFS - Hadoop Framework Internal Steps
  2. What is HDFS
  3. HDFS Federation in Hadoop Framework
  4. HDFS High Availability
  5. NameNode, DataNode And Secondary NameNode in HDFS

You may also like -

File Read in HDFS - Hadoop Framework Internal Steps

In this post we’ll see what all happens internally with in the Hadoop framework when a file is read in HDFS.

Reading file in HDFS

With in the Hadoop framework it is the DFSClient class which communicates with NameNode and DataNodes. The instance of DFSClient is created by DistributedFileSystem which is the implementation class in case of HDFS.

When client application has to read a file it calls open() method on DistributedFileSystem which in turn calls open() method of DFSClient. DFSClient creates an instance of DFSInputStream which communicates with NameNode.

DFSInputStream connects to the NameNode and gets the location of first few blocks of the file. Note that default replication factor is 3 so, for every block, information about 3 DataNodes that are storing the specific block will be sent by NameNode. In the list sent by NameNode, DataNodes are also ordered by their proximity to the client for each block. So client application will try to read data from a local DataNode first rather than the remote DataNode.

Reading blocks from DataNodes

Once the list of blocks is retrieved client application calls read on the wrapper stream FSDataInputStream. In turn the wrapper stream DFSInputStream which already has a list of DataNodes connects to the nearest DataNode storing the first block of the file and start streaming data to the client. DFSInputStream will follow the same procedure for all the blocks in the list; connect to DataNode storing that block, stream data, disconnect from the DataNode.

Since NameNode sends only the first few blocks of the file, DFSInputStream also communicates with NameNode to get the DataNode information for the next set of blocks. This process will continue until all the blocks of the file are read.

Once all the blocks are read and streamed to the client, stream is closed.

In the architecture followed for reading the file the client is directly connecting and getting the data from DataNodes. No data flows through NameNode.

In case of any error while reading the block another DataNode storing the same block is tried. That is where replication helps.

HDFS data flow for file read in HDFS

That's all for this topic File Read in HDFS - Hadoop Framework Internal Steps. If you have any doubt or any suggestions to make please drop a comment. Thanks!


Related Topics

  1. File Write in HDFS - Hadoop Framework Internal Steps
  2. What is HDFS
  3. HDFS Federation in Hadoop Framework
  4. HDFS High Availability
  5. NameNode, DataNode And Secondary NameNode in HDFS

You may also like -

Thursday, 22 February 2018

HDFS High Availability

This post gives an overview of HDFS High Availability (HA), why it is required and how HDFS High Availability can be managed.

Problem with single NameNode

To guard against the vulnerability of having a single NameNode in a Hadoop cluster there are options like setting up a Secondary NameNode to take up the task of merging the FsImage and EditLog or to have an HDFS federation to have separate NameNodes for separate namespaces. Still, having a backup NameNode is something that was missing and the NameNode was a single point of failure (SPOF) in an HDFS cluster. This impacted the total availability of the HDFS cluster in two major ways:
  1. In the case of an unplanned event such as a machine crash, the cluster would be unavailable until an operator restarted the NameNode.
  2. In case of NameNode planned maintenance for any software or hardware upgrades would result in windows of cluster downtime.

In these cases a new NameNode will be able to start service requests only after-

  1. Loading the FsImage into memory and merging all the transactions stored in EditLog.
  2. Getting enough block reports from DataNodes as per configuration to leave safemode.
In a large cluster this may mean a lapse of close to half an hour where the whole cluster remains idle.

In Hadoop 2.x release a new feature HDFS High Availability is introduced that addresses the above problems by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration.

HDFS High Availability architecture

With HDFS high availability two separate machines are configured as NameNodes in a cluster.

Out of these two NameNodes, at any point in time, exactly one of the NameNodes is in an Active state and responsible for all client operations in the cluster.

The other NameNode remains in a Standby state. It has to maintain enough state to provide a fast failover if necessary.

For the standby NameNode to keep its state synchronized with the Active node both nodes should have access to an external shared entity. For this shared access there are two options-

  • Quorum Journal Manager (QJM)
  • Shared NFS directory

General concept in both of these options is same whenever any namespace modification is performed by the Active node, it logs a record of the modification to the shared access too. Standby node reads those edits from the shared access and applies them to its own namespace.

That way both the Namenodes are synchronized and standby NameNode can be promoted to the Active state in the event of a failover.

Both of the Namenodes should also have the location of all blocks in the Datanodes. To keep that block mapping information up-to-date DataNodes are configured with the location of both NameNodes, and send block location information and heartbeats to both.

Shared access with NFS

If you are using NFS directory as shared access then it is required that both the NameNodes have access to that NFS directory.

Any namespace modification performed by the Active node is looged to edit log file stored in the shared directory. The Standby node is constantly watching this directory for edits, and as it sees the edits, it applies them to its own namespace.

Shared access with QJM

In case of QJM both nodes communicate with a group of separate daemons called “JournalNodes” (JNs). When any namespace modification is performed by the Active node, it durably logs a record of the modification to a majority of these JNs.

The Standby node is capable of reading the edits from the JNs, and is constantly watching them for changes to the edit log. As the Standby Node sees the edits, it applies them to its own namespace.

There must be at least 3 JournalNode daemons, since edit log modifications must be written to a majority of JNs. This will allow the system to tolerate the failure of a single machine. You may also run more than 3 JournalNodes, but in order to actually increase the number of failures the system can tolerate, you should run an odd number of JNs, (i.e. 3, 5, 7, etc.). Note that when running with N JournalNodes, the system can tolerate at most (N - 1) / 2 failures and continue to function normally.

Configuration for HA cluster

The configuration changes required for HA NameNodes are as follows. Changes are required in hdfs-site.xml configuration file.

dfs.nameservices – Choose a logical name for this nameservice, for example “mycluster”

<property>
  <name>dfs.nameservices</name>
  <value>mycluster</value>
</property>

Then you need to configure the NameNodes suffixed with the nameservice ID. If the individual ids of the Namenodes are namenode1 and namenode2 and the nameservice ID is mycluster.

<property>
  <name>dfs.ha.namenodes.mycluster</name>
  <value>namenode1,namenode2</value>
</property>

You also need to provide fully-qualified RPC address and fully-qualified HTTP address for each NameNode to listen on.

<property>
  <name>dfs.namenode.rpc-address.mycluster.namenode1</name>
  <value>machine1.example.com:8020</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.namenode1</name>
  <value>machine1.example.com:50070</value>
</property>


<property>
  <name>dfs.namenode.rpc-address.mycluster.namenode2</name>
  <value>machine2.example.com:8020</value>
</property>
<property>
  <name>dfs.namenode.http-address.mycluster.namenode2</name>
  <value>machine2.example.com:50070</value>
</property>
To provide the location of the shared storage.

In case of QJM with 3 machines.

<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>qjournal://node1.example.com:8485;node2.example.com:8485;
      node3.example.com:8485/mycluster</value>
</property>
In case of NFS
<property>
  <name>dfs.namenode.shared.edits.dir</name>
  <value>file:///mnt/filer1/dfs/ha-name-dir-shared</value>
</property>

dfs.client.failover.proxy.provider.[nameservice ID] - the Java class that HDFS clients use to contact the Active NameNode. The two implementations which currently ship with Hadoop are the ConfiguredFailoverProxyProvider and the RequestHedgingProxyProvider. So use one of these unless you are using a custom proxy provider.

<property>
  <name>dfs.client.failover.proxy.provider.mycluster</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

Handling NameNode failover in HDFS HA

In case of failover the Standby node should be promoted to active state where as the previously active NameNode should transition to standby mode.

That failover can be managed manually by using the following command.

hdfs haadmin -failover

This command will initiate a failover between two NameNodes. If the first NameNode is in the Standby state, this command simply transitions the second to the Active state without error. If the first NameNode is in the Active state, an attempt will be made to gracefully transition it to the Standby state.

For automatic failover ZooKeeper can be configured. In ZooKeeper there is a ZKFailoverController (ZKFC) process which monitors and manages the state of the NameNode.

Each of the machines which runs a NameNode also runs a ZKFC. The ZKFC pings its local NameNode on a periodic basis with a health-check command.

If the node has crashed, frozen, or otherwise entered an unhealthy state, the health monitor will mark it as unhealthy. In that case failover mechanism is triggered.

Fencing procedure in HDFS High Availability

In a Hadoop cluster, at any given time, only one of the NameNode should be in the active state for the correctness of the system. So, it is very important to ensure that the NameNode that is transitioning from active to standby is not active any more.

That is why there is a need to fence the Active NameNode during a failover.

Note that when using the Quorum Journal Manager, only one NameNode is allowed to write to the edit logs in JournalNodes, so there is no potential for corrupting the file system metadata. However, when a failover occurs, it is still possible that the previous Active NameNode could serve read requests to clients.

There are two methods which ship with Hadoop: shell and sshfence.

sshfence - SSH to the Active NameNode and kill the process. In order for this fencing option to work, it must be able to SSH to the target node without providing a passphrase.

<property>
      <name>dfs.ha.fencing.methods</name>
      <value>sshfence</value>
</property>

<property>
      <name>dfs.ha.fencing.ssh.private-key-files</name>
      <value>/home/exampleuser/.ssh/id_rsa</value>
</property>

shell - run an arbitrary shell command to fence the Active NameNode. The shell fencing method runs an arbitrary shell command. It may be configured like so:

<property>
      <name>dfs.ha.fencing.methods</name>
      <value>shell(/path/to/my/script.sh arg1 arg2 ...)</value>
</property>

Reference : https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

That's all for this topic HDFS High Availability. If you have any doubt or any suggestions to make please drop a comment. Thanks!


Related Topics

  1. What is HDFS
  2. Replica Placement Policy in Hadoop Framework
  3. What is SafeMode in Hadoop
  4. Introduction to Hadoop Framework

You may also like -

What is SafeMode in Hadoop

When the NameNode starts in a Hadoop cluster, following tasks are performed by NameNode.

  1. NameNode reads the FsImage and EditLog from disk, applies all the transactions from the EditLog to the in-memory representation of the FsImage, and flushes out this new version into a new FsImage on disk.
  2. Receive block reports from the DataNodes in the cluster.

While these tasks are performed NameNode stays in a state known as Safemode in Hadoop.

Restrictions during Safemode

While NameNode is in Safemode no write operations can be performed by any client application. Only read only operations like listing the files in a directory are allowed to work during that period.

If NameNode doesn’t wait till it gets enough block reports from the DataNodes in the cluster it will start replicating the blocks again to the DataNodes after startup. So it is important that the NameNode stays in the Safemode until the stated tasks are not finished.

When does NameNode exit SafeMode

After a percentage (Which is configurable) of safely replicated data blocks checks in with the NameNode (plus an additional 30 seconds), the NameNode exits the Safemode state automatically.

Configuration for Safemode

Configuration parameters for safemode are as follows, these configuration parameters are in configuration file hdfs-site.xml.

dfs.namenode.safemode.threshold-pct - Specifies the percentage of blocks that should satisfy the minimal replication requirement defined by dfs.namenode.replication.min parameter. Values less than or equal to 0 mean not to wait for any particular percentage of blocks before exiting safemode. Values greater than 1 will make safe mode permanent. Default value if 0.999f or 99.9% which means 99.9% of block should satisfy the minimal replication requirement.

dfs.namenode.safemode.extension - Determines extension of safe mode in milliseconds after the threshold level is reached. Default value is 30000 miliseconds or 30 seconds.

dfs.namenode.safemode.min.datanodes - Specifies the number of datanodes that must be considered alive before the name node exits safemode. Values less than or equal to 0 mean not to take the number of live datanodes into account when deciding whether to remain in safe mode during startup. Values greater than the number of datanodes in the cluster will make safe mode permanent. Default value is 0 so by default number of live datanodes is not taken into account.

HDFS commands for safemode

If required, HDFS could be placed in Safemode explicitly using bin/hdfs dfsadmin -safemode command. NameNode front page UI also shows whether Safemode is on or off.

1- Entering the safemode - hdfs dfsadmin -safemode enter

2- Leaving the safemode - hdfs dfsadmin -safemode leave

3- Checking whether Namenode is in safemode - hdfs dfsadmin -safemode get

4- If you want any file operation command to block till HDFS exists safemode - hdfs dfsadmin -safemode wait

5- Forcefully exit the safemode - hdfs dfsadmin -safemode forceExit

That's all for this topic What is SafeMode in Hadoop. If you have any doubt or any suggestions to make please drop a comment. Thanks!


Related Topics

  1. What is HDFS
  2. Replica Placement Policy in Hadoop Framework
  3. HDFS Federation in Hadoop Framework
  4. HDFS High Availability
  5. Introduction to Hadoop Framework

You may also like -

Wednesday, 21 February 2018

HDFS Federation in Hadoop Framework

In this post we’ll talk about the HDFS Federation feature introduced in Hadoop 2.x versions. With HDFS federation we can have more than one NameNode in the Hadoop cluster each managing a part of the namespace.

HDFS architecture limitations

HDFS follows a master/slave architecture where NameNode acts as a master and then there are several DataNodes. NameNode in the HDFS architecture performs the following tasks.

  1. Managing the Namespace – NameNode manages the file system namespace. NameNode stores metadata about the files and directories in the file system.

    NameNode also supports all the namespace related file system operations such as create, delete, modify and list files and directories.

  2. Block Storage Service - NameNode also maintains block mapping i.e. in which DataNode any given block is stored.

    NameNode Supports block related operations such as create, delete, modify and get block location.

In the prior HDFS architecture (Before Hadoop 2.x) only single namespace for the entire cluster is allowed and a single Namenode manages the namespace. In a large clusters with many files having a single NameNode may become a limiting factor for memory needs and scaling.

HDFS Federation tries to address this problem.

HDFS Federation support for multiple NameNodes/NameSpaces

HDFS Federation addresses this limitation of having a single NameNode by adding support for multiple Namenodes/namespaces to HDFS. In HDFS Federation architecture NameNodes manages a part of the namespace. Thus HDFS Federation adds support for namespace horizontal scaling.

As Example – If there are two namespaces /sales and /finance then you can have two Namenodes; NameNode1 and NameNode2 where NameNode1 manages all files under /sales and NameNode2 manages all files under /finance.

In HDFS federation Namenodes are federated; the Namenodes are independent and do not require coordination with each other. Thus, a Namenode failure does not prevent the Datanode from serving other Namenodes in the cluster.

Note that Datanodes are still used as common storage for blocks by all the Namenodes. Each Datanode registers with all the Namenodes in the cluster. Datanodes send periodic heartbeats and block reports.

HDFS Federation Namespace Volume

In HDFS Federation a set of blocks that belong to a single namespace is known as Block Pool.

A Namespace and its block pool together are called Namespace Volume. It is a self-contained unit of management.

HDFS Federation configuration

If we take the same example of having two namespaces /sales and /finance and two namenodes NameNode1 and NameNode2 then the required configuration changes are as follows.

You need to add the dfs.nameservices parameter to your configuration file (hdfs-site.xml) and configure it with a list of comma separated NameServiceIDs.

<property>
    <name>dfs.nameservices</name>
    <value>sales,finance</value>
</property>

You also need to add following configuration parameters suffixed with the corresponding NameServiceID.

<property>
    <name>dfs.namenode.rpc-address.sales</name>
    <value>namenode1:8020</value>
</property>
<property>
    <name>dfs.namenode.http-address.sales</name>
    <value>namenode1:50070</value>
</property>
<property>
    <name>dfs.namenode.rpc-address.finance</name>
    <value>namenode2:8020</value>
</property>
<property>
    <name>dfs.namenode.http-address.finance</name>
    <value>namenode2:50070</value>
</property>
You can use ViewFs to create personalized namespace views. That change is required in core-site.xml.
<property>
  <name>fs.defaultFS</name>
  <value>viewfs:///</value>
</property>
Then the mount table config variables for sales and finance.
<property>
    <name>fs.viewfs.mounttable.default.link./sales</name>
    <value>hdfs://namenode1:8020/sales</value>
</property>

<property>
    <name>fs.viewfs.mounttable.default.link./finance</name>
    <value>hdfs://namenode2:8020/finance</value>
</property>

Reference: Whttps://hadoop.apache.org/docs/r2.9.0/api/org/apache/hadoop/fs/viewfs/ViewFs.html

That's all for this topic HDFS Federation in Hadoop Framework. If you have any doubt or any suggestions to make please drop a comment. Thanks!


Related Topics

  1. What is HDFS
  2. Replica Placement Policy in Hadoop Framework
  3. Introduction to Hadoop Framework

You may also like -

Friday, 16 February 2018

NameNode, DataNode And Secondary NameNode in HDFS

HDFS has a master/slave architecture. With in an HDFS cluster there is a single NameNode and a number of DataNodes, usually one per node in the cluster.

In this post we'll see in detail what NameNode and DataNode do in Hadoop framework. Apart from that we'll also talk about Secondary NameNode which can take some of the work load of the NameNode.

NameNode in HDFS

The NameNode is the centerpiece of an HDFS file system. NameNode manages the file system namespace by storing information about the file system tree which contains the metadata about all the files and directories in the file system tree.

Metadata stored about the file consists of file name, file path, number of blocks, block Ids, replication level.

This metadata information is stored on the local disk. Namenode uses two files for storing this metadata information.

  • FsImage
  • EditLog

We’ll discuss these two files, FsImage and EditLog in more detail in the Secondary NameNode section.

NameNode also keeps in it’s memory location of the DataNodes that store the blocks for any given file. Using that information Namenode can reconstruct the whole file by getting the location of all the blocks of a given file.

Client application has to talk to NameNode to add/copy/move/delete a file. Since block information is also stored in NameNode so any client application that wish to use a file has to get BlockReport from NameNode. The NameNode returns list of DataNodes where the data blocks are stored for the given file.

DataNode in HDFS

Data blocks of the files are stored in a set of DataNodes.

Client application gets the list of DataNodes where data blocks of a particular file are stored from NameNode. After that DataNodes are responsible for serving read and write requests from the file system’s clients. Actual user data never flows through NameNode.

The DataNodes store blocks, delete blocks and replicate those blocks upon instructions from the NameNode.

DataNodes in a cluster periodically send a blockreport to the NameNode too. A blockreport contains a list of all blocks on a DataNode.

Secondary NameNode in HDFS

Secondary NameNode is more of a helper to NameNode, it is not a backup NameNode server which can quickly take over in case of NameNode failure.

Before going into details about Secondary NameNode let’s go back to the two files which were mentioned while discussing NameNode – FsImage and EditLog.

  • EditLog – All the file write operations done by client applications are first recorded in the EditLog.
  • FsImage – This file has the complete information about the file system metadata when the NameNode starts. All the operations after that are recorded in EditLog.

When the NameNode is restarted it first takes metadata information from the FsImage and then apply all the transactions recorded in EditLog. NameNode restart doesn’t happen that frequently so EditLog grows quite large. That means merging of EditLog to FsImage at the time of startup takes a lot of time keeping the whole file system offline during that process.

Now you may be thinking only if there is some entity which could take over this job of merging FsImage and EditLog and keep the FsImage current that will save a lot of time. That’s exactly what Secondary NameNode does. Its main function is to check point the file system metadata stored on NameNode.

The process followed by Secondary NameNode to periodically merge the fsimage and the edits log files is as follows-

  1. Secondary NameNode gets the latest FsImage and EditLog files from the primary NameNode.
  2. Secondary NameNode applies each transaction from EditLog file to FsImage to create a new merged FsImage file.
  3. Merged FsImage file is transferred back to primary NameNode.

The start of the checkpoint process on the secondary NameNode is controlled by two configuration parameters which are to be configured in hdfs-site.xml.

  • dfs.namenode.checkpoint.period - This property specifies the maximum delay between two consecutive checkpoints. Set to 1 hour by default.
  • dfs.namenode.checkpoint.txns - This property defines the number of uncheckpointed transactions on the NameNode which will force an urgent checkpoint, even if the checkpoint period has not been reached. Set to 1 million by default.

Following image shows the HDFS architecture with communication among NameNode, Secondary NameNode, DataNode and client application.

NameNode and DataNode in HDFS

Reference -

That's all for this topic NameNode, DataNode And Secondary NameNode in HDFS. If you have any doubt or any suggestions to make please drop a comment. Thanks!


Related Topics

  1. What is HDFS
  2. Introduction to Hadoop Framework
  3. Replica Placement Policy in Hadoop Framework
  4. What is SafeMode in Hadoop

You may also like -

Wednesday, 14 February 2018

Replica Placement Policy in Hadoop Framework

HDFS as the name says is a distributed file system which is designed to store large files. A large file is divided into blocks of defined size and these blocks are stored across machines in a cluster. These blocks of the file are replicated for reliability and fault tolerance. For better reliability Hadoop framework has a well defined replica placement policy.

Rake aware replica placement policy

Large HDFS instances run on a cluster of computers that commonly spread across many racks so rack awareness is also part of the replica placement policy.

If two nodes placed in different racks have to communicate that communication has to go through switches.

If machines are on the same rack then network bandwidth between those machines is generally greater than the network bandwidth between machines in different racks.

HDFS replica placement policy

Taking rank awareness and fault tolerance into consideration the replica placement policy followed by Hadoop framework is as follows-

For the common case, when the replication factor is three

  1. Put one replica on the same machine where the client application (application which is using the file) is, if the client is on a DataNode. Otherwise choose a random datanode for storing the replica.
  2. Store another replica on a node in a different (remote) rack.
  3. The last replica is also stored on the same remote rack but the node where it is stored is different.

In case replication factor is greater than 3, for the first 3 replicas policy as described above is followed. From replica number 4 onwards node location is determined randomly while keeping the number of replicas per rack below the upper limit (which is basically (replicas - 1) / racks + 2).

HDFS Replication pipelining

While replicating blocks across DataNodes, pipelining is used by HDFS. Rather than client writing to all the chosen DataNodes data is pipelined from one DataNode to the next.

For the default replication factor of 3 the replication pipelining works as follows-

The NameNode retrieves a list of DataNodes that will host the replica of a block. Client gets this list of 3 DataNodes from NameNode and writes to the first DataNode in the list. The first DataNode starts receiving the data in portions, writes each portion to its local storage and then transfers that portion to the second DataNode in the list. The Second DataNode follows the same procedure writes the portion to its local storage and transfers the portion to the third DataNode in the list.

For replication factor of 3 following image shows the placement of replicas.

HDFS replica placement

Reference : http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html#Data_Replication

That's all for this topic Replica Placement Policy in Hadoop Framework. If you have any doubt or any suggestions to make please drop a comment. Thanks!


Related Topics

  1. What is HDFS
  2. Introduction to Hadoop Framework

You may also like -