25. How building this Raspberry Pi Cluster helped me in my cloud/security journey ☁🔐🍓🥧!

Feb 2, 2025 5 min read technical

While I worked at Google during the publishing of this post / video, the views expressed here are my own and may not reflect those of my employer. Only publicly available material is used to put together the content of this article and video. The website and the Youtube videos are NOT monetized.

I have always loved teaching. More than a decade ago, I’d conduct training sessions for Hadoop, Kafka, Storm etc. - Not as huge as AI today, but “Big data” was Big then.

I come mostly from a software/security background, so whenever I’d talk to Chris and Jason at the office and they’d talk about racks and servers and stuff like that, I’d feel like an outsider!

So, one of these days - I decided to build myself a full fledged home lab and deploy as many of big data applications on it as possible and take my training sessions to the next level.

Initially, I purchased 4 Raspberry Pi 2 Model B and the micro SD cards needed (as their “hard drives”) but that was just a start, I ended up getting the following -

  1. A rack to mount these machines
  2. A micro-usb power source to power all of them at once
  3. A switch for the layer two connectivity among them (Pi did not have a wifi chip then)
  4. A mini wifi router to act as bridge between my main wifi router which was sitting in a different room and local network of these 4 machines by acting as a bridge
  5. Lan cables (some fancy ones!)

And with that I had the hardware setup ready.

How did this setup work?

alt

Well so,

  1. Each Raspberry Pi had 32 Gb of storage.
  2. Each one had a class C static IP private addresses
  3. When connecting with each other they’d connect via switch
  4. The bridge acted as an internet gateway and connected the machines to internet.

I deployed Hadoop across these 4, got Kafka and Storm going as well (almost a decade now, so not sure about the exact details and this post will be super long if I elaborate!).

And then my training sessions were conducted on a “real” setup. So for example - I could show fault tolerance of the HDFS by switching off a node etc.

But, more so than the Big data, which I realized I don’t really enjoy so much - I was enjoying putting together this setup.

And of course that meant soon, mission creep followed.

I added another node so that I could do even more on this setup -

  1. I started running my website on this setup (thanks Verizon for the relatively stable static IPV4)
  2. Ran my VPN server (openVPN) so that I could login to my home network from anywhere. why? you ask? - I say why not!
  3. Built a NAS (network attached storage) of 64 GB on a USB stick attached to a powered USB hub and created a backup (rsync) of the same on another USB (both bought from best buy in Black Friday deals)

Then, at some point, I realized that the machines are throttling due to heat, so I added heat sinks on the processors - did not help much. So added a USB fan to the setup (RHS image) - then it was ok. 🔥🔥 ✇✇✇✇ !!!

I also learnt that the bridge was not the best in terms of latency, so this time I plugged the switch directly into the router.

alt

I had already started working on cloud around this time and realized it came very naturally to me, especially after the home lab setup!!

  1. I could relate so easily with IaaS (Infrastructure as a Service) which involved concepts like VMs, File systems, VPCs, NAT Gateways, Firewalls etc.
  2. I knew how to deploy a web application all the way from purchasing a domain to actually getting the application deployed and served to users
  3. Knew how to load balance
  4. Knew the concepts and importance of backing up and how to schedule backups
  5. How to be frugal and save on costs and run things only when they are needed because I was running a relatively low powered commodity setup
  6. How to solve latency issues and other issues like heat (not particularly something I’d encounter on cloud but it’s important to know - machines throttle when pushed to capacity).
  7. I could easily relate to concepts of clustering having run Big data (relatively) successfully on my setup
  8. I was always a shell/bash fan and learnt so many new commands through the process!
  9. Most importantly, I learnt fair bit of networking and how to run a cluster on the internet securely!!

I cleared my AWS associate and professional certifications and officially got into cloud full time.

The learnings from setting up my own (nano or even pico) “data center”, come in very handy even today.

In fact, cloud security which is what I do at Google now has its foundations in failing and learning in my home lab.

There are 2 reasons I wanted to write this today

  1. A lot of people are mesmerized by the power of AI and want to learn AI. Here’s my take on it - Unless your day job is AI (training, fine tuning models etc) - learning about AI is not as useful as using AI for your core work (system design, coding, testing, DevOps, security etc.). And being comfortable with your core work requires you to - Actually Work in your core area. Fail and learn from those failures and Fix things and solve problems.
  2. And, I came across the following video by Jeff Greeling . And something inside me wants to go and build something, again! After all, Raspberry Pis are far more powerful today than 10 years ago. May not be running models locally but may be an ethical hacking rig! May be!!



Keep building, fail, fix, solve, learn and repeat - That’s the journey towards getting better at your core work! 👋

Thank you for reading through, Please like 👍, share 🔗 and comment ✍ if you found it useful.

PS: I know it’s a strawberry emoji in the title. There is no emoji for Raspberry so had to choose between 🍒🍓🍇!!!

-Nikhil

comments powered by Disqus