How to provide Elasticity to the DataNode Storage Using LVM
Integrating LVM with Hadoop and providing Elasticity to DataNode Storage.
Today, I am going to show you how to provide elasticity to the DataNode.
What is LVM?
LVM stands for Logical Volume Management. It is a concept for the storage. Many storage device which provides elasticity follows LVM concept.
LVM helps to provide the Elasticity to the storage Device.
What is Elasticity?
Suppose I have a external device of 10 GB where My server is runnning, but after some days If I am able to add or remove some storage space then this concept is known as Elasticity.
Many companies prefer the storage devices which follows elasticity concept. So that in future if they need to add some more space on the runtime they can easily add without any interruption.
Hadoop doesn’t support elasticity so we integrate Hadoop concept with LVM concept.
Now, let’s see how to integrate LVM Concept with Hadoop DataNode
- Setup a master Node and DataNode, I already setup my Master Node and Data Node on AWS Cloud.
- Add a external Volume of 20GB for LVM management. Below image shows all the harddisk attach to the ec2-instance.
Command used: fdisk -l to check all the connected device.
3. Now, first we have to create a physical Volume(PV). Before creating PV, we have to install lvm2.
Command used: yum install lvm2
Now, we have to create PV. Below image shows the PV creation.
Command Used: pvcreate /dev/xvdf
For display the Pvs we can use pvdisplay command.
4. Now, we have to create Volume Group(VG) so that further we can create a partion of 10GB from Volume group. Below is the image of creation of VG.
Command used: vgcreate myhadoopvg /dev/xvdf
Command used: vgdisplay to show the volume group details.
5. No, we are ready to create a logical Volume partition from volume group. Below is the image to show the creation of Logical Volume partiton.
Command used: lvcreate — size 10G — name myhadooplv muhadoopvg
lvdisplay command is used to display the LV(logical Volume)
6. Format the LV partion and then we mount on the mount point.
Mount point will be our datanode storage folder (i.e /dn5). Below is the image shows the formatting and mounting.
Command used to format the partition: mkfs.ext4 /dev/myhadoopvg/myhadooplv
Now fdisk -l command shows the formatted drive , see the below image.
Now, we are going to mount to our precreated folder(/dn5).
See the below images properly.
Now, by using df -h command, we can see that our mounting has sucessfully done. See the below image
Note: Now, our DataNode contribute 10GB (9.78GB) to the Master Node. See the below image for proof.
Command used: hadoop dfsadmin -report
GUI view of Hadoop Cluster
Now, suppose our 10GB limit will exhaust by storing the data, now we can extend the limit of 10 GB to 20 GB on the runtime. How? because we are using LVM comcept.
Let’s see how can we extend the size of 10GB to 15GB.
7. We need to increase the LV size from VG, I am increasing +5GB more. Below is the image which shows the increament from 10 G to 15 G.
command used: lvextend — size +5G /dev/myhadoopvg/myhadooplv
8. We have to do one more thing to use complete 15 Gb. We have to format 5Gb extra part, so we are resize2fs tool, to format the Non-partition portion from the complete LV.
Command used:resize2fs /dev/myhadoopvg/myhadooplv
9. Now, we did all the things, let’s see that our disk size is increased or not.
See the below image, it shows that our disk size has increased from 10GB to 15 GB.
Command used: fdisk -l
GUI view of Hadoop Cluster
CLI View of hadoop Cluster
Our task has done, we are succesfully increased the size of logical partition from 10GB to 15GB as per our requirment. Logical Volume Management Concept(LVM) is so easy to implement. It is one of the best way to provide the Elasticity to the Physical Volume.
Thanks for Reading, Give one clap to encourage me.