Raspberry Pi Hadoop Cluster

I am currently in the process of learning Hadoop architecture, administration and the MapReduce programming model. I started reading about Hadoop and took free online courses but there is something missing. I wanted to try out what I read or what I was told on those training. In some exercises in the training a VM was used as a single node Hadoop server. But for me it doesn’t make sense, so I tried out setting up more VM and configured them into a Hadoop cluster. But still the experience was not very satisfying because it lacks the touch of the hardware. I wasn’t really running a cluster but simply just a bunch of virtual machines connected together in a virtual network. So I thought “How about I build a cluster of cheap computers.” In fact, that’s what Hadoop was designed for, to run on a cluster of commodity hardware. I tried googling around about cheap cluster computer and found a few blogs and videos about guys who ran MPI and Hadoop on a mini cluster of Raspberry Pi boards. So I decided to build my own mini cluster of Raspberry Pis. After all, the best way to learn new things is to get your hand dirty.

There are a couple of Raspberry Pi boards laying around my desk at home. I used them in some experiments before. But I needed more, so I ordered 3 more of these boards. Raspberry Pi is a S$50 single board computer with ARM7 1Ghz  dual-core CPU, 1GB of RAM and a microSD slot for storage.

I also bought SD cards and, USB power cables, an 8-port network switch and a high-current USB power supply/charger. Luckily I was able to find all off them locally in the neighbourhood shops except for the Raspberry Pi which I ordered online and received the next day.

I started thinking about how to clump this boards together in a rack-like structure where it is easy to cable them up to a network switch. I found few ideas online about using some stand-off bolts and nuts and acrylic boards to stack them up together. The problem is I don’t have these materials so I started walking around the house to come up with ideas and to look for materials. The mounting holes in the Raspberry Pi are 2mm wide so I started looking for bolt and screws of this size but didn’t find any. Then I went to the laundry area where I found a wire-made clothes hanger. The wire core metal is around 2mm diameter covered with a PVC plastic insulator around 1 mm thick. I grabbed it and decided to build a mini rack out of it. I created a half round loop and literally stitched the boards together.

The network switch I bought is powered by a 5V power supply at 600mA so the USB power supply can actually power it.
I downloaded a Linux distro called Raspbian Jessie it is a lightweight variant of Debian linux intended for headless Raspberry Pi server. It’s a stripped down version of Raspbian Wheezy without the UI. I then updated it with latest libraries and installed Java 8 JDK.

I followed the steps by that was described in this blog but with some extra steps to configure a second interface card with a Wi-Fi dongle and voila. I now have a Hadoop cluster running.

I later bought a set of acrylic stackable case to make the rack more stable and look good. I stripped out the board of the cheap network switch (I bought online for $7 from china) and mounted it the one of the acrylic stackable case so it will look uniform. Now I am hadooping with this little beast. I later upgraded to Hadoop 2 and installed Apache spark for future Apache Spark experiments and adventures.


  1. You are MacGyver with a laundry trick 😋 ! I wish you keep us updated with real life home hardware experience 'cause so far doing non performing Hadoop helloword with VM is frustrating ! Congrats for this achievement !

  2. Hi, I am really happy to found such a helpful and fascinating post that is written in well manner. Thanks for sharing such an informative post.R Programming Online Training | Hadoop Online Training

  3. Hai admin,,,,,,,,,,,,your blog was informative.I read your all blog.so keep blogging.Thanks for sharing. If you want to reach high level............... Android Training in Chennai
    Dot Net Training in Chennai
    Hadoop Training in Chennai
    Selenium Training in Chennai

  4. I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.

    Hadoop Online Training
    Data Science Online Training

  5. Thanks for one marvelous posting! I enjoyed reading it; you are a great author. I will make sure to bookmark your blog and may come back someday. I want to encourage that you continue your great posts
    Hadoop Training in Bangalore

  6. Informative post about hadoop, i am looking forward for realtime hadoop online training institute.

  7. A debt of gratitude is in order for one grand posting! I appreciated understanding it, you are an awesome creator. I will try to bookmark your blog and may return sometime in the future. I need to energize that you proceed with your extraordinary posts

  8. Excellent blogs.We have to learning that the lot new information.I will be like this information.

  9. I am perusing your post from the earliest starting point, it was so intriguing to peruse and I feel on account of you for posting such a decent blog.
    Article Submission sites | Technology | Education | Daily Consumer Life

  10. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Good discussion. Thank you.
    Six Sigma Training in Abu Dhabi
    Six Sigma Training in Dammam
    Six Sigma Training in Riyadh

  11. Some us know all relating to the compelling medium you present powerful steps on this blog and therefore strongly encourage contribution from other ones on this subject while our own child is truly discovering a great deal.
    safety course in chennai

  12. Played on BGAOC with big wins? NOT? Come to us as soon as possible and win with us. perfect all slots casino Come get your chance to win.

  13. Качественный декоративный алюминиевый led профиль я обычно беру в компании Ekodio, советую.

  14. Home Mart is a site about Home Improvement, Furniture, Home Appliances and many more.
    Check out the best
    furniture nz
    furniture sale
    Dog Cages

  15. I would definitely thank the admin of this blog for sharing this information with us. Waiting for more updates from this blog admin.
    salesforce Training in Bangalore
    uipath Training in Bangalore
    blueprism Training in Bangalore

  16. Wow...What an excellent informative blog, really helpful. Thank you so much for sharing such a wonderful article with us.keep updating..
    aws Training in Bangalore
    python Training in Bangalore
    hadoop Training in Bangalore
    angular js Training in Bangalore
    bigdata analytics Training in Bangalore

  17. Nice article i was really impressed by seeing this article, it was very interesting and it is very useful for me.I get a lot of great information from this blog. Thank you for your sharing this informative blog..

    Big Data Hadoop Training In Chennai | Big Data Hadoop Training In anna nagar | Big Data Hadoop Training In omr | Big Data Hadoop Training In porur | Big Data Hadoop Training In tambaram | Big Data Hadoop Training In velachery

  18. Excellent post and it is really useful for most of the freshers.
    Fence Accessories

  19. Good Post! , it was so good to read and useful to improve my knowledge as an updated one, keep blogging.After seeing your article I want to say that also a well-written article with some very good information which is very useful for the readers....thanks for sharing it and do share more posts likethis. https://www.3ritechnologies.com/course/aws-online-training/

  20. Good Post! , it was so good to read and useful to improve my knowledge as an updated one, keep blogging.After seeing your article I want to say that also a well-written article with some very good information which is very useful for the readers....thanks for sharing it and do share more posts likethis. https://www.3ritechnologies.com/course/aws-online-training/

  21. I am glad to post a worthy article about the German Language Course and IELTS Coaching from KCR consultants, this may change your career growth and language skill.

  22. Thanks For Sharing such a wonderful article the way you presented is really amazing
    Best Software Training Institutes

  23. Very well written blog post. Thanks for sharing with us.
    Best Web development company in Hyderabad



Architecting Systems Like A Rock Band

In Software, orchestration often means control, synchronization, mediation and scheduling of decoupled application services in order to ful...