February 20th, 2019 by Brian Beeler
Mistakes Were Made: Home Lab Horror Stories
In partnership with /r/homelab we gave away a Synology RS214 recently. The mission was simple, just post an anecdote about that one time you made a mistake in your home lab. There were over 150 replies with stories that ranged from funny to depressing. Below are a few of our and the community's favorites.
Apparently you need to be careful when using rm on the command line.
“rm -rf / directory/path/“ wasn’t quite what I wanted.
So.... my wife and I are police officers so we both have that type A personality. One night she gives me the go ahead to play games and lab around while she is watching HGTV (on our smart TV via Sling). Well while waiting for my friends to hop on, I'm doing my usual goof around and changing some settings on the AP to try and fix some congestion issues.
Insert Morgan Freeman voice "This was not the smartest idea Scat_Pack71 ever had and would quickly regret it"
So I go ahead and reboot the AP without thinking and the DNS server as well, apparently this was at the very crucial moment of a couple picking their BeachFront Bargain.....which my wife was trying to win at a contest with herself.
It was like time went into slow motion but instant screaming as I clicked the button. The amount of "f***s" I heard followed by "I told you to game first before you f*** with our network" "God you're like the department's IT rebooting the dispatch computers while we're working"
Not my smoothest move ever but thankfully she forgave me and promptly returned the favor by plugging in a wireless mouse when I wasn't looking and distracted by dogs...followed by a day of screwing with me with the old phantom mouse routine.
Best was upgrading the firmware on my router while my fiancee was taking an online test that was due very shortly after causing the network to go down. I was sweating a lot on that one hoping it would come back up with no issues. General rule is I only do the upgrades when she isn't home now.
Oh my favorite has to be when I was developing on AWS for homelab and then going to work the next day and not using the correct AWS API key... Ended up having a 50k AWS bill hit me until I realized I was on my own instances...
Always a fun day at the office when you have to explain how bad that is. Luckily I didn't deploy anything crazy out there and just testing new apps.
Early on in my homelabbing career, I decided it would be great to host my own everything: Email, websites, Pi-Hole DNS, SSH remote access, RDP, you name it. I also thought it would also be great to use the default ports and no VPN for everything because why would anyone care what I am hosting. Needless to say, it didn't take long for a port scanner to find out, and boy were they in for a treat.
How I wish i did off-site backup. House burned down. Lost everything, but iCloud photos. Only to realize, never enabled :(
This would allow me to finally achieve, off site backup.
The one time in my homelab of creating two datastores in Exsi but not realizing that one was in a separate raid. Deleting everything, I mean everything. I lost over 4tb of data storage. Lesson learned, I've documented everything from day one now. :(
That one time when you got a shiny secondhand dl360p-g8 and see that it has ilo advanced in the post screen. But being secondhand you don't have the credentials to said ilo, so you "reset it to default". Not knowing that doing so deletes your ilo advanced license as well, great design feature there.
September 2004. I had been in my own place for just over a year. Was working steadily. Student loans paid off. I had started my own home lab with scrounged parts from work. That year was an especially bad hurricane season and two storms moved up the east coast a week after one another dumping tons of rain in my area. I drove home and was surprised to see a lot of roads partially submerged.
Anyway, I got home, ate dinner, showered, and went to my upstairs desktop to start messing around installing an Ubuntu Warty beta on an old PowerEdge - you know the purplish-charcoal ones that were all plastic? This was before the days of free VMware server for Linux, ESXi, HyperV, etc. I wanted to start using Apache on Linux at work instead of IIS on Windows Server 2003. Anyway, I had installed it the day before I was SSHing into the server to start configuring Apache and PHP and MySQL. This was my first foray into LAMP installations. After a while, I started noticing connection retries to package servers when I was installing packages. Then, my connection totally drops. I walk down to the basement and see water on the floor. I thought a pipe had burst. My switch was on the floor next to my cable modem and old Compaq Presario desktop running IPCop (I couldn't afford a rack and was too cheap to buy a card table). The area where I lived was doing a lot of construction so they had messed up the drainage enough to get it clogged from all the rain. This caused the sewers to back up and the water came up through the floor drain in my basement. Que hours of frantic trips up and down stairs carrying equipment and other things. After all was said and done, I had 18 inches of standing water in my basement. Lost the IPCop desktop, cable modem, some spare hard drives, and a bunch of other non-computer related stuff. What hurt the most was my golden cartridge copy of Ocarina of Time was damaged by the water. It was a fun time dealing with insurance with all that stuff because I had to provide purchase information - hard to do with scrounged work equipment.
From /u/Nero8762 (contest "winner")
16 years ago when my ex and I split up, I scanned over 4k pics of the kids growing up, over a 2 month period (scanning). She got the originals, I got the digital copies.
2008, surge/power outage from storm fries my computer and HDD. No back ups of pics.
Ex passes from cancer in 2009. Her husband can't find the originals after going thru everything. Years of memories gone.
I know, it's not an embarrassing story, but a lesson to your all. Have a back up of your back up of your back ups. Practice 1-2-3.
StorageReview fully embraces the home lab community. If you want to share your home lab excellence or shame with us, please send an email to firstname.lastname@example.org.