Lost arrays on HP array controller.

[RimBlock]

Hi,

So I have been playing with trying to get my server to boot from the HP Proliant SPP update image. All attempts have failed as the USB stick was not seen as bootable.

On rebooting my server I get the message that the array configuration on my P812 controller is invalid and the controller cache is not configured.

Booting in to the ACU reports all drives are at unassigned good status and all my logical arrays have vanished.

My biggest concern is my two raid 5 arrays (3 & 5 drive). Whilst they are not critical, it would be a pain to get all the data back.

I have tried removing the drives and rebooting and then inserting the drives after booting in to the ACU but they are still detected as unconfigured good.

My business data is on a raid 1 array on a different controller and so is not affected.

Any hope of self recovery ?.

Thanks
RB

 

[RimBlock]

Thanks for the reply.

Hopefully that may work. I did try with a raid 0 array of 3 SSD drives (fast scratch disk) but as the drives had just been configured before the config was lost, they didn't have anything on them so not possible to tell . Creating the array did not suddenly show up a logical drive. I had to recreate that as well.

The two big raid 5 arrays I know have data but I would most like to get that data back. I may try with the one holding my VMs as I can always rebuild the VMs where as the data drive will be more difficult to recover all the data back.

RB

 

[RimBlock]

Did you try starting up the server without the cache module installed and then shutting it down again and putting the cache module back in. I recently had some issues with the a storage controller and this is what resolved the problem. Not sure if it would fix your problem though but might be worth a try.

The cache module on a storage controller is the part at the back that you can remove like an addon component.

Not for that problem......

As per Ringihos suggestion I just recreated one of the arrays and then all of the arrays suddenly reappeared. I lost the raid 5 array with my VMs on but that was due to my own mistake and not a biggie as a couple of the VMs had backups on other drives and the main one I had been thinking about rebuilding anyway.

I have hit another bigger issue now though...

I updated the firmware on the controller (I think) via the SSP for Oct 2010 and HP bootable update key drive. I say think as after booting and getting the SPP update first prompt (i.e. SPP found, do you want to run it) I went and did a check on some info I wanted to hand before proceeding. On my return to the server, it had booted in to the SPP update and was extracting the exe file with all the patches. Ho Hum I though, popped off and get a drink and the machine had rebooted. The problem is that the machine now only boots, goes in to the P812 Initilisation boot screen, reports the number of logical arrays and then instantly reboots the server. I have tried pulling all the drives, changing motherboard slots, pulling the cache and capacitor but all to no effect (the car cannot even be detected in the first slot on my S1200BTL Intel board any more).

So now I have two raid 5 arrays inaccessible and a server I cannot boot past the P812 initilisation screen.

Unless anyone has any great ideas I can only see the following options;

1. Get an old Proliant, ML110 G7 or MiniServer (card is too long to fit with the cover on) and try reflashing. Failing that try to RMA to HP who will probably need an HP server serial before they will be willing to do anything (if they are anything like IBM).
2. Get a replacement main board for the P812 (quite a few floating about on EBay US) but risk of having the same problem with vanishing array configs and possible inability to flash.
3. Get a P412 to enable me to pull the data off the arrays (claimed they can read the same arrays as the P812) and then get an LSI card.

1 is possibly the cheapest for an old Proliant (possibly) but no guarantee of a reflash working or HP being willing to support.

2. Is most likely to work but expensive @ around US$550 and will potentially just put me back in the pre-flash position.

3. Is probably the most expensive as I would probably go for a decent LSI / Arecca card.

Any ideas...

RB

Remember, I get myself in to these expensive, frustrating and downright unbelievable situations so you don't have too .

 

[RimBlock]

When you say SSP 2010 do you mean SPP 2012? There's a brand new one (October 2012), so it's worth trying that.

Yep it was 10/2012 (HP_Service_Pack_for_Proliant_2012.10.0-0_713293-001_spp_2012.10.0-SPP2012100.2012_1005.37). It did the auto update though so I didn't see the update applied or if there was any issue.

I have to assume that you don't have ProLiant hardware, and my experience with Smart Array controllers is that they are EXTREMELY picky about hardware. In fact, I have never been able to get my P410i to even be recognised by anything that isn't a ProLiant (it has only ever worked in my Microserver and my ML110 G7).

I have asked around in the office but am not too hopeful.

A ML110 G7 is a possibility and I could probably justify the cost for my home buisness but it was something I didn't want to have to purchase at this point.

Based on all your troubles, you may have a hardware compatibility problem...

That I could but if it was not working on the Intel server board fine for around a month before it lost the config. Interestingly the new firmware fixes an issue where the controller can loose its configuration now and then.

Ultimately, if you haven't got ProLiant hardware, my recommendation would be to sell the P812 (you'll get good money for it) and buy an LSI or something like that.

I do tend to find I end up 'pushing the boundaries' sometimes without realising it. The card had worked fine up until it was flashed but the flash seems to have done something that now makes it incompatible with my Intel board. That is frustrating to say the least.

I was glad it was working but then....

Disaster!

My only suggestion would be to try the update again from the latest SPP. You might find it better! Maybe just try running the one update.

Now just for the sake of it I put it in my MSI LGA775 board I use for flashing M1015 controllers and was shocked to find it booted. I reset it and it booted again just fine. It does not seem to detect the drives in the DAS attached externally but I can get back in to the SSP USB boot drive again.

I have reflashed the controller and am just taking a look now before retrying it back on the Intel server.

One thing I have noticed is that I have no longer got the F8 key option on boot to go in to the config firmware on boot. This maybe an option I need to set though and so I have gone in to the ACU to have a look around.

RB

 

[RimBlock]

Possibly I could get it working on the Intel board but I ended up so fed up I got a ML110 G7 (E3-1220, 2x500GB, 4GB ram).

Put the controller in and used the SSP on the server to get everything to the latest patch level and now it will not see the external drives in my Supermicro chassis.

It had no problem in the MSI board but now in the HP server it just cannot see anything attached externally. I have put a WD drive in the bay in the front of the server and it seems fine (detected etc) and the 2x 500GB drives are just WD RE4s (not HP branded).

I have also tried connecting the drives up through the cabling but via individual drives rather than through the Supermicro backplane (which worked fine previously). No change.

Really pulling my hair out now. I like a challenge but this is getting ridiculous and expensive.

RB

 

[RimBlock]

Just to try out I took the SFF-8088 cable to a PCI ext -> int and then the SFF-8087 -> 4xSATA and it still didn't work.

I then tried connecting directly to the internal connector on the controller and all 4 drives were detected straight away.

I have tried both of my dual port int -> ext SAS PCI dual brackets and no change. I guess both could be faulty .....

RB

 

[RimBlock]

Possible I guess but they look like they are just traces from int to ext plugs with no chips etc on the PCI brackets.

The brackets are here.

RB

 

[RimBlock]

That was one of the reasons behind getting the ML110 .

Ok, some more results.
I have flashed back to Jan 2012 firmware and have tested all the external cables. They are all working and the drives are seen. Of course as soon as I tried testing I noticed my Supermicro chassis witht eh 8 drive hotswap used for my P812 arrays had powered off and wouldn't restart. A PSU swap and swapback and a little more tinkering and it then came on again.

THe controller now sees all the external drives in the Supermicro chassis (the chassis was on and working fine prior to back flashing the card).

I have tidied up the ML110 G7, closed it up and put it back in the rack but now when I put the power in the fans go straight to high speed and the single blue light comes on the front panel. I am now unable to start the server.

I clearly irritated the gods in a past life..... .

I am now trying to look up why the fans would hit top speed and only the front blue light would come on.

RB

 

[RimBlock]

Little update.

The P812 still will not go in the Intel board even after regressing the firmware. I have put it back in the MSI LGA775 board and it is working fine. I will build this, in the sort term at least, as a SAN with dual nics.

The warranty on the ML110 G7 is next business day (Tuesday) for replacement. They are likely to replace the system board.

My problem now is that I can see both of my arrays on the controller and in vSphere but my main data array it will not import without formatting even though it is reporting there is a VMFS logical drive on the array.

I can stand to loose the data and start off as a SAN from scratch with that space but ..... If I can recover it then I would be much happier I think.

RB

 

[RimBlock]

You have had a hard time! It's good to see you have managed to get it sorted. I'm extremely tempted by an ML110. Is it good? Ignoring the broken bit

I am fairly impressed with the package. As someone who builds servers fro clients usually around Supermicro and Intel parts, I can see the package is fairly good quality and the software that comes with it is pretty good.

I done have much confidence in the SPP updates at the moment though.

The E3-1220 costs around 500 quid over here with 4GB ram and 2x500GB sata WD RE4s. I got it a bit cheaper as a HP reseller but there are no cash back offers.

Support over web chat has been reasonable so I will have to see how the engineer does with the replacement.

I think I would still recommend it as a very nice package. It can be a little noisey as others have commented in the ML110 thread and it only takes 4 drives but is a nice starter package for a small business or avid home user.

RB

 

[RimBlock]

I am really considering sorting out a home lab type machine for ESXi. I want to learn some more on Server 2012, Exchange and Lync.

It really is very easy and makes great use of old hardware. I was shocked to see just how little processor resourse WHS 2011 was taking on my server which prompted me to try virtulisation, from OpenVZ on CentOS -> ProxMox and Xen and finally on to vSphere. One of the best choices I have made in a while.

It seems that we are in similar job roles! I work for a managed hosting company and have built quite a few servers (looking at 1700+ now!). Over a thousand were Intel chassis based but we've recently moved over to HP kit (much better).

Sort of. Finance Enterprise IT where the focus is more on reliability plus my own business doing IT solutions (HP/IBM/Custom) for SMEs so more entry level servers and networking plus a few home user setups. Working on a Wifi Mesh solution for a client at the moment which is quite interesting. Multiple access points with only 1 SSID and the devices are handed off to each access point depending on range seamlessly. Not cheap though.

Note to self: never allow RimBlock anywhere near my computer gear! =)

Haha, that a bit harsh .

To be fair, this Array controller has caused me the biggest headache in a long time.

It has got me to finally setup a basic SAN though and that seems to be running pretty nicely although I still have some more work to do on it.

HP engineer is booked to visit tonight to replace the MB and possibly the RAM.

RB

 

[RimBlock]

Well the ML110 has been repaired. They changed the MB & the ram but believed it is a faulty power switch motherboard connection. I don't think they were convinced.

I have just found that the ML110 G7 works with the same Kingston ram as my Intel S1200BTL so I have taken half of the 32GB in the Intel based server and put it in the ML110 G7 along with the original 4GB and all is looking good. I will put SBS 2011 on it natively and keep the P812 in the MSI SAN box for now as it appears to be working fine like that. My Intel server will become my test and home VMs server (Minecraft, SABnzb etc).

RB