Skip to main content

Raspberry Pi Hadoop Cluster


I am currently in the process of learning Hadoop architecture, administration and the MapReduce programming model. I started reading about Hadoop and took free online courses but there is something missing. I wanted to try out what I read or what I was told on those training. In some exercises in the training a VM was used as a single node Hadoop server. But for me it doesn’t make sense, so I tried out setting up more VM and configured them into a Hadoop cluster. But still the experience was not very satisfying because it lacks the touch of the hardware. I wasn’t really running a cluster but simply just a bunch of virtual machines connected together in a virtual network. So I thought “How about I build a cluster of cheap computers.” In fact, that’s what Hadoop was designed for, to run on a cluster of commodity hardware. I tried googling around about cheap cluster computer and found a few blogs and videos about guys who ran MPI and Hadoop on a mini cluster of Raspberry Pi boards. So I decided to build my own mini cluster of Raspberry Pis. After all, the best way to learn new things is to get your hand dirty.

There are a couple of Raspberry Pi boards laying around my desk at home. I used them in some experiments before. But I needed more, so I ordered 3 more of these boards. Raspberry Pi is a S$50 single board computer with ARM7 1Ghz  dual-core CPU, 1GB of RAM and a microSD slot for storage.





I also bought SD cards and, USB power cables, an 8-port network switch and a high-current USB power supply/charger. Luckily I was able to find all off them locally in the neighbourhood shops except for the Raspberry Pi which I ordered online and received the next day.

I started thinking about how to clump this boards together in a rack-like structure where it is easy to cable them up to a network switch. I found few ideas online about using some stand-off bolts and nuts and acrylic boards to stack them up together. The problem is I don’t have these materials so I started walking around the house to come up with ideas and to look for materials. The mounting holes in the Raspberry Pi are 2mm wide so I started looking for bolt and screws of this size but didn’t find any. Then I went to the laundry area where I found a wire-made clothes hanger. The wire core metal is around 2mm diameter covered with a PVC plastic insulator around 1 mm thick. I grabbed it and decided to build a mini rack out of it. I created a half round loop and literally stitched the boards together.


The network switch I bought is powered by a 5V power supply at 600mA so the USB power supply can actually power it.
I downloaded a Linux distro called Raspbian Jessie it is a lightweight variant of Debian linux intended for headless Raspberry Pi server. It’s a stripped down version of Raspbian Wheezy without the UI. I then updated it with latest libraries and installed Java 8 JDK.

I followed the steps by that was described in this blog but with some extra steps to configure a second interface card with a Wi-Fi dongle and voila. I now have a Hadoop cluster running.





I later bought a set of acrylic stackable case to make the rack more stable and look good. I stripped out the board of the cheap network switch (I bought online for $7 from china) and mounted it the one of the acrylic stackable case so it will look uniform. Now I am hadooping with this little beast. I later upgraded to Hadoop 2 and installed Apache spark for future Apache Spark experiments and adventures.

Comments

  1. You are MacGyver with a laundry trick 😋 ! I wish you keep us updated with real life home hardware experience 'cause so far doing non performing Hadoop helloword with VM is frustrating ! Congrats for this achievement !

    ReplyDelete
  2. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post.R Programming Online Training | Hadoop Online Training

    ReplyDelete
  3. Hai admin,,,,,,,,,,,,your blog was informative.I read your all blog.so keep blogging.Thanks for sharing. If you want to reach high level............... Android Training in Chennai
    Dot Net Training in Chennai
    Hadoop Training in Chennai
    Selenium Training in Chennai

    ReplyDelete
  4. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.

    Hadoop Online Training
    Data Science Online Training

    ReplyDelete
  5. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts
    Hadoop Training in Bangalore

    ReplyDelete
  6. Informative post about hadoop, i am looking forward for realtime hadoop online training institute.

    ReplyDelete

Post a Comment

Popular

DIY Airsoft Chrono [PC-based]

Chrono or Chronograph, the term used by Airsofter and Paintball players to refer to the device that measures the muzzle velocity of firearms. This is also called MVMD, short for Muzzle Velocity Measuring Device by the British Army. Muzzle Velocity is the velocity of the bullet as it exits the barrel of the firearm.

A year ago I was an active Airsofter in Iligan who's more interested in modifying and improvising AEGs (Automatic Electric Gun) rather than playing airsoft. I have made my AEG susceptible to high current burns by using locally available MOSFETs. I have improved velocity by replacing cylinder gaskets and spring. I have increased the rounds per minute by altering the armature windings of the drive motor. And most of all, I have made my own cheap alternative PC-based Chrono.

I have posted an artivle before that details the development of my chrono but unfortunately the online forum was hacked/deleted by webmaster of proboards.com due to some violation in the contents. So, …

Disable PrintScreen on C# without Keyboard Hooks

Yes, there is a simple solution to prevent grabbing information on your C# application screen with the keyboard's printscreen key without using keyboard hooks or calling COM interops. The solution makes use of Windows Forms Message Filter to trap keyboard events on your application window.

To trap keyboard events with Windows Message Filter, you need to implement the IMessageFilter interface and override the member PreFilterMessage(ref Message WM) method. This is the method called whenever a Form receives a keyboard or mouse event. You may want to read more about IMessageFilter.

The problem is that even though you have trapped the PrintScreen keypress event, the captured image will still persist to the clipboard. Therefore the simplest solution is to clear up the clipboard right after the PrintScreen is pressed on the keyboard.

This is how the overridden method will look like this:

publicbool PreFilterMessage(refMessage WM)
{
Keys kCode = (Keys)…

DIY Mic Shock Mount

For the past couple of weeks I was looking for a cheap microphone shock mount and the cheapest I saw that was available in my country was around US$160 so I decided to build a cheap one. I only spent less than $6 and a 2 hours of my not very precious time.Before I left office today, bought materials and tools to for my DIY shock mount which are a piece of 4”-diameter PVC pipe coupling ($0.50), a set of hair ties($0.40), set of stove bolts ($0.50), a set of coping saw frame and blades ($2.5).First, I decided to cut the PVC coupling so it would look like Rode SM3 shock mount’s frame.Then I used the section of the PVC coupling that was removed to be a attachment brace to the original mic holders base and bolted it to the base of the frame. I had to heat that part so could bend it to a desired angle.And then cut slots on the edges for the rubber suspensions (hair ties). Right after positioning the suspensions, i found a problem. The rubber bands slip every time i attempted to position the…