Skip to main content

Raspberry Pi Hadoop Cluster

I am currently in the process of learning Hadoop architecture, administration and the MapReduce programming model. I started reading about Hadoop and took free online courses but there is something missing. I wanted to try out what I read or what I was told on those training. In some exercises in the training a VM was used as a single node Hadoop server. But for me it doesn’t make sense, so I tried out setting up more VM and configured them into a Hadoop cluster. But still the experience was not very satisfying because it lacks the touch of the hardware. I wasn’t really running a cluster but simply just a bunch of virtual machines connected together in a virtual network. So I thought “How about I build a cluster of cheap computers.” In fact, that’s what Hadoop was designed for, to run on a cluster of commodity hardware. I tried googling around about cheap cluster computer and found a few blogs and videos about guys who ran MPI and Hadoop on a mini cluster of Raspberry Pi boards. So I decided to build my own mini cluster of Raspberry Pis. After all, the best way to learn new things is to get your hand dirty.

There are a couple of Raspberry Pi boards laying around my desk at home. I used them in some experiments before. But I needed more, so I ordered 3 more of these boards. Raspberry Pi is a S$50 single board computer with ARM7 1Ghz  dual-core CPU, 1GB of RAM and a microSD slot for storage.

I also bought SD cards and, USB power cables, an 8-port network switch and a high-current USB power supply/charger. Luckily I was able to find all off them locally in the neighbourhood shops except for the Raspberry Pi which I ordered online and received the next day.

I started thinking about how to clump this boards together in a rack-like structure where it is easy to cable them up to a network switch. I found few ideas online about using some stand-off bolts and nuts and acrylic boards to stack them up together. The problem is I don’t have these materials so I started walking around the house to come up with ideas and to look for materials. The mounting holes in the Raspberry Pi are 2mm wide so I started looking for bolt and screws of this size but didn’t find any. Then I went to the laundry area where I found a wire-made clothes hanger. The wire core metal is around 2mm diameter covered with a PVC plastic insulator around 1 mm thick. I grabbed it and decided to build a mini rack out of it. I created a half round loop and literally stitched the boards together.

The network switch I bought is powered by a 5V power supply at 600mA so the USB power supply can actually power it.
I downloaded a Linux distro called Raspbian Jessie it is a lightweight variant of Debian linux intended for headless Raspberry Pi server. It’s a stripped down version of Raspbian Wheezy without the UI. I then updated it with latest libraries and installed Java 8 JDK.

I followed the steps by that was described in this blog but with some extra steps to configure a second interface card with a Wi-Fi dongle and voila. I now have a Hadoop cluster running.

I later bought a set of acrylic stackable case to make the rack more stable and look good. I stripped out the board of the cheap network switch (I bought online for $7 from china) and mounted it the one of the acrylic stackable case so it will look uniform. Now I am hadooping with this little beast. I later upgraded to Hadoop 2 and installed Apache spark for future Apache Spark experiments and adventures.


  1. You are MacGyver with a laundry trick 😋 ! I wish you keep us updated with real life home hardware experience 'cause so far doing non performing Hadoop helloword with VM is frustrating ! Congrats for this achievement !

  2. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post.R Programming Online Training | Hadoop Online Training

  3. Hai admin,,,,,,,,,,,,your blog was informative.I read your all keep blogging.Thanks for sharing. If you want to reach high level............... Android Training in Chennai
    Dot Net Training in Chennai
    Hadoop Training in Chennai
    Selenium Training in Chennai

  4. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.

    Hadoop Online Training
    Data Science Online Training

  5. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts
    Hadoop Training in Bangalore

  6. Informative post about hadoop, i am looking forward for realtime hadoop online training institute.

  7. A debt of gratitude is in order for one grand posting! I appreciated understanding it, you are an awesome creator. I will try to bookmark your blog and may return sometime in the future. I need to energize that you proceed with your extraordinary posts

  8. Excellent blogs.We have to learning that the lot new information.I will be like this information.

  9. I am perusing your post from the earliest starting point, it was so intriguing to peruse and I feel on account of you for posting such a decent blog.
    Article Submission sites | Technology | Education | Daily Consumer Life

  10. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Good discussion. Thank you.
    Six Sigma Training in Abu Dhabi
    Six Sigma Training in Dammam
    Six Sigma Training in Riyadh

  11. Some us know all relating to the compelling medium you present powerful steps on this blog and therefore strongly encourage contribution from other ones on this subject while our own child is truly discovering a great deal.
    safety course in chennai

  12. Played on BGAOC with big wins? NOT? Come to us as soon as possible and win with us. perfect all slots casino Come get your chance to win.

  13. Качественный декоративный алюминиевый led профиль я обычно беру в компании Ekodio, советую.

  14. Home Mart is a site about Home Improvement, Furniture, Home Appliances and many more.
    Check out the best
    furniture nz
    furniture sale
    Dog Cages

  15. I would definitely thank the admin of this blog for sharing this information with us. Waiting for more updates from this blog admin.
    salesforce Training in Bangalore
    uipath Training in Bangalore
    blueprism Training in Bangalore


Post a Comment


DIY Airsoft Chrono [PC-based]

Chrono or Chronograph, the term used by Airsofter and Paintball players to refer to the device that measures the muzzle velocity of firearms. This is also called MVMD, short for Muzzle Velocity Measuring Device by the British Army. Muzzle Velocity is the velocity of the bullet as it exits the barrel of the firearm.

A year ago I was an active Airsofter in Iligan who's more interested in modifying and improvising AEGs (Automatic Electric Gun) rather than playing airsoft. I have made my AEG susceptible to high current burns by using locally available MOSFETs. I have improved velocity by replacing cylinder gaskets and spring. I have increased the rounds per minute by altering the armature windings of the drive motor. And most of all, I have made my own cheap alternative PC-based Chrono.

I have posted an artivle before that details the development of my chrono but unfortunately the online forum was hacked/deleted by webmaster of due to some violation in the contents. So, …

Disable PrintScreen on C# without Keyboard Hooks

Yes, there is a simple solution to prevent grabbing information on your C# application screen with the keyboard's printscreen key without using keyboard hooks or calling COM interops. The solution makes use of Windows Forms Message Filter to trap keyboard events on your application window.

To trap keyboard events with Windows Message Filter, you need to implement the IMessageFilter interface and override the member PreFilterMessage(ref Message WM) method. This is the method called whenever a Form receives a keyboard or mouse event. You may want to read more about IMessageFilter.

The problem is that even though you have trapped the PrintScreen keypress event, the captured image will still persist to the clipboard. Therefore the simplest solution is to clear up the clipboard right after the PrintScreen is pressed on the keyboard.

This is how the overridden method will look like this:

publicbool PreFilterMessage(refMessage WM)
Keys kCode = (Keys)…

DIY Mic Shock Mount

For the past couple of weeks I was looking for a cheap microphone shock mount and the cheapest I saw that was available in my country was around US$160 so I decided to build a cheap one. I only spent less than $6 and a 2 hours of my not very precious time.Before I left office today, bought materials and tools to for my DIY shock mount which are a piece of 4”-diameter PVC pipe coupling ($0.50), a set of hair ties($0.40), set of stove bolts ($0.50), a set of coping saw frame and blades ($2.5).First, I decided to cut the PVC coupling so it would look like Rode SM3 shock mount’s frame.Then I used the section of the PVC coupling that was removed to be a attachment brace to the original mic holders base and bolted it to the base of the frame. I had to heat that part so could bend it to a desired angle.And then cut slots on the edges for the rubber suspensions (hair ties). Right after positioning the suspensions, i found a problem. The rubber bands slip every time i attempted to position the…