When you shut down the server and it doesn't come back up...

[jaybee]

IT staff have all been there. Here's an odd one though. I shut down my home server (which is basically a virtual gaming PC / server hybrid) for the first time in ages. I changed the CPU, the GPU and added an additional M2 nvme drive. Booted back up and all I got was the classic fans spin for a split second, then shut down. No further pressing of the power button would do anything until turning off at the wall and attempting again.

What follows is a list of troubleshooting steps until I finally arrived at the problem. So where's your money going? Which component?

This machine is stacked and fully populated with 10 fans, 8 HDDs, 2 ssds, 2 nvmes, 2 GPUs, 1 hba card so it could be a minefield but....I only changed the above 3 things.

I start by reseating and then removing entirely the new gpu. Same.
I remove both GPUs since I had to remove the second one to get to the second M2 slot and could have not seated it correctly. Same.
I start to worry it's the new CPU so put the old one back in. Same.
I reset cmos. Same.
I remove the new second M2 nvme. Same.
I try a spare PSU and connect it just to CPU and main mobo power headers quickly....it posts! But I don't have GPUs plugged in at this point.
I start to plug more stuff into the PSU and after a few quick cycles to see if it would keep posting, it has the same issue again...
I go back to the original psu and plug in only the motherboard and CPU power....it posts!
I realize it's not the PSU and might be something is causing it difficulty like a hdd or sata power cable maybe...
I gradually plug stuff in until I find that one of the power cables with 2 X sata and 2 X molex is the problem.
I trace it down to a single molex connected to power fans...
The molex has been stretched and knocked possibly overtime and the black and red pins are making barely a connection and are sticking out the back of the plastic connector block slightly causing a short.

Once again molex proves to be the worst cable connector type we still have to live with, closely followed by micro usb!

Do you still touch bare metal servers today? Any interesting/funny stories?

 

[Rroff]

Not a fan of Molex connectors but I've never really had a problem with them being as I'm slightly "OCD" about making sure they are connected right.

Personally do like micro USB - not always the most robust if people lack for mechanical sympathy though.

Had a fun one but it will take some explaining.

Was setting up a new server and while doing so was doing some maintenance on an existing system to make things easier I had each system hooked up to a monitor but system 1 was connected to both monitors while system 2 just connected to one of them.

Someone, not me, had been lazy and not put the cover back on the PDU power button designed to prevent this kind of thing - while connecting up system 2 to the KVM I accidentally leaned on the power button...

In the oh **** moment I didn't notice I'd dislodged the keyboard connection to the KVM for system 1.

I powered on the live system and nothing... no display on the monitor at all, no lights on the KVM, didn't think at the time to check link activity for the network.

Absolute head scratcher, took the whole thing apart and nothing.

Eventually realised that because it had 2 monitors connected it was booting up with the secondary display as the primary monitor, so no display on the main monitor, but the secondary display's input was set to system 2 and the KVM was disconnected so no keyboard lights, etc. for system 1 and because of the unexpected shutdown it was booting back into filesystem verification so no services were booting back up.

Not sure if that is clear enough what happened :s

EDIT: Irony the system I was setting up was to be the failover for system 1...

 

[Berserker]

I'm not a fan of Molex connectors either but that sounds like the pins weren't inserted properly in the first place.

Not had any major hardware disasters other than shorting a PSU with a dodgy fan controller. I have had a PSU and a UPS go bang spectacularly (the UPS tripped a breaker for the entire building!), but that was hardware faults, not my fault.

Software, on the other hand, that has happened (deleting customer files, fortunately not critical ones).

 

[Quartz]

So where's your money going? Which component?

New CPU? I would make sure the BIOS was up to date before changing the CPU. This is a particular problem with Ryzen CPUs.

 

[Fluffybob]

Ruddy hate Molex plug, I've even cable tied them before now