full image - Repost: Homelab oddities (from Reddit.com, Homelab oddities)
Mining:
Exchanges:
Donations:
Hey everyone!I had some really strange stuff happen in my home lab. I legitimately cannot explain how/why.Bare with me here.....Got an email from my systems monitoring server saying that one one of my DCs DNS service was not running, and the monitoring system couldn't restart the service.Weird... so I remoted in, and try to manually restart the DNS Service. No dice. Reboot the DC... no dice. I figured the DC was tosted, so i began spinning up a new VM to replace it. (I always keep 3, for paranoia reasons)After i judt promoted the new DC, my email just gets flooded with alerts of random servers and services not being able to be reached from VMs on one paticular host.Ok. Something is strange. I attempt to shut all the servers off on that host (some through ESXI console since RDP was intermittent at this time.)Some of the servers couldn't power off. Just got stuck in a "shutting off" state. I decided to VMotion the surviving DCs off to another host in my cluster and force the host to power down through iLO console.But we aren't over quite yet......Then my SAN sends an alert email and text message saying the SAN management and communications are now offline. All my VMs (except one DC stored locally on the host for JUST this scenario) hard stop. Ended up needing to hard reboot the SAN since it was no longer communicating. Once that was up, I brought the host back up, get it back in the cluster and all VMs spun up just fine. Even the initial problematic DC came up, with all services running normally.It's been a few hours and all my systems are happily chugging along. I ran some malware scans across my network, checked Nmap to see if there was a remote possibility that 2 systems shared the same IP... ALL came back normal.I'm beyond baffled. I'm happy I make almost everything redundant.Anyone got weird stories of stuff you just can't explain why/how?I'd love to hear it!
Social Media Icons