Multiple switches failed!?
Wondering if anyone has ever seen anything like this or has any ideas what could cause this.
We have an office that has a total of 6 Meraki MS350-48 switches. They are in two stacks in two separate locations. Two switches in one stack upstairs connected to an APC UPS with an MX and some servers. The other 4 switches are in a stack in a warehouse connected to a PDU in a rack. Two CAT6 runs connect the two stacks.
Late last week we suddenly had the site go down and started troubleshooting. It turned out one of the switches in the first stack was down and simply would not even boot up - power supplies cables etc all good but switch won't even power on. We move the uplink to the other switch but it won't link up - like no link light even. The MX is on and all good - reachable, hasn't gone down etc... We notice some of the link lights on the switch seem weird - like the right half of the switch has some link lights on on ports with no cables and some ports won't link up at all. But roughly port 1-24 seem to have normal link lights so we move the uplink to port 24 and voila it links up and gets uplink. We also move the downlink that goes to the other stack to one of these ports and it also lights up. So of these two switches one is dead and one has half the ports completely failed. We even troubleshoot with Meraki on the phone and this is the result. No firmware updates or power outages seem to have occurred.
So this isn't even the end of the story - now the other stack. Similar story - the entire stack is offline - switch-1 has dead ports including the stacking ports. Switch 2 and 3 are the same. Switch 4 is offline and won't even power on. We end up replacing all 4 switches in the second stack as well to get everything back online.
Out of six switches, two are completely dead and four others have ports dead we can't get to work - even after reset. They were in two separate locations and some of them were on a UPS. We've had the occasional switch fail before but how can 6 switches all fail at the same time? Nothing else at the site seems to have had any issues.