Patching 2 year old downed servers that have duplicates on the domain?

Bleek · 9 Jun 2009 at 10:58

We have a pair of DR Exchange servers, they are exact replicas of the two live Exchange servers and attach to the same databases on SAN storage.

These two DR boxes have been down for two years and require bringing up-to-date via patching, the question is how best we do it?

We can't just power on these servers because they share computer names with the live pair and don't want to disrupt the production Exchange environment.

To make matters more difficult they are Blades and we must patch via PatchLink software, but that requires network access and has no 'offline' facility - or so our PatchLink expert tells us.

Any ideas?

Bleek · 9 Jun 2009 at 11:05

That does sound rather simple... isn't it always the case you miss the simplest of options?

However won't that change the SIDs and cause problems in the future if we need to failover to them?

Bleek · 9 Jun 2009 at 13:29

mr.bond said:
Talk to whichever software vendor sold you the HA solution and ask them for a patching process, assuming you've purchased a third party app.

We use Neverfail and have a very specific p[rocess which must be followed for this exact scenario you've desribed.

It was a contractor and a third party, they don't mention patching just how to fail over to the servers.

I agree though, there should have been a patching plan because having them sat there for years is very amateurish (before you say anything it's just been dumped on my plate).

Bleek · 9 Jun 2009 at 13:29

mr.bond said:
That wouldn't be reccomended Neil. Can elaborate if you wish.

Please do, seriously it would help weigh up the pros and cons.

Bleek · 10 Jun 2009 at 11:55

Can I not disable network, boot up off domain, logon locally, enable network and just use Windows Update?

Exchange 2003, Windows 2003.

As requested the documented DR process is as follows (in reduced form from a 35 page word document):

The blade server chassis houses 2 blade servers, these have been produced by taking images of the live servers and deploying the images to the recovery servers. The only exceptions to the image on the recovery servers is their IP addresses, Default Gateway, Domain / Workgroup membership and the machine Security Identifier (SID)

After imaging the disaster recovery servers will be reconfigured as follows;

1. Boot the server
2. Log on as a local Administrator

Disable the ‘Local Area Network Connection’
3. Go to ‘Start’, ‘Control Panel’, Network Connections’, right click the ‘Team 1’ connection and select ‘Disable’

4. Reconfigure the servers as follows;

Server Name : CSLWINEXE01
Workgroup : EXE_RECOVERY
IP Address : ##removed
Subnet Mask : ##removed
Default Gateway : ##removed
DNS Server : ##removed

Server Name : CSLWINEXE02
Workgroup : EXE_RECOVERY
IP Address : ##removed
Subnet Mask : ##removed
Default Gateway : ##removed
DNS Server : ##removed

5. Reboot the server when prompted, log on as a ‘Local Administrator’

6. Change the server SID using the Microsoft ‘NEWSID’ utility, NEWSID can be downloaded from;
http://www.microsoft.com/technet/sysinternals/Security/NewSid.mspx

7. When prompted select ‘Random SID’

8. The server will reboot when NEWSID has completed

NOTE : The SID change operation is only required to completed once during the initial configuration of the Disaster Recovery servers.

Active Directory

The recovery servers are members of a Workgroup ‘EXE_RECOVERY’ and therefore it is necessary to add the recovery servers into the Domain. The ‘Computer Account’ in Active Directory must be ‘reset’ prior to bringing the recovery servers on-line, to do this proceed as follows;

NOTE : Under NO circumstances must the Computer object be deleted from Active Directory, doing so will require the re-installation of the Server Operating System and Exchange.

1. Log onto the nearest Domain Controller as a ‘Domain Administrator’

2. Go to ‘Start’, ‘Administrative Tools’, Active Directory Users and Computers’

3. Expand the Domain object ‘###, expand the ‘Secure’, select the ‘Secure Servers’ object.

4. From the list in the right hand window locate the required server name (CSLWINEXE01 or CSLWINEXE02), right click on it and select ‘Reset Account’ as shown below;

#SNIP

The recovery servers have the same name as the live servers but have different IP addresses, it is therefore necessary to change the DNS records for the servers so that the recovery servers can be located.

#SNIP

Enable networking on both Exchange servers.

That is from what I can see the process for getting the two servers up, obviously there is loads of other info about SAN, databases, Exhcange services etc etc but obviously we don't care for that here.

Bleek · 10 Jun 2009 at 13:16

That is the process for failover from Head office the datacentre to DR datacentre.

There is no 'patching' method in this document.

Thanks for your help so far.

Bleek · 10 Jun 2009 at 15:41

Ok well we tried it the dirty way, boot the DR Exchange box with network disabled (it's not a member of the domain by the way), logged in locally, enabled network and gave it an IP (it noticed a duplicate name on the network but production was unaffected), gave it a route to the proxy and it can access MSoft updates... so I'm guessing this will work?

It found SP2 as it's first patch, nice... and out-of-date.

Can you see any reasons why doing it this way would/could cause a problem?

Bleek · 11 Jun 2009 at 09:23

I'm a Project Manager with a technical background but by no means an Exchange engineer and to my knowledge this company has never had one on site. Our current Windows team know basic admin but that aside, no one has come forward about improving the current situation.

We don't want to DR failover as we're not confident it'll be clean and comfortable, we can't afford that kind of disruption for now, a full DR test is later in the year.

isoToxin said:
Argh. Never nice when you're left to pick up the pieces on a poorly designed system.

Tell me about it!

Is there any reason why you couldn't just have these DR boxes on your network as active hosts with unique names/SIDs? In a DR scenario, you could just mount the information store onto one of them and re-home everyone's mailboxes to the new storage group.

That does sound like an option to look at.

Looking to the future, if you chose to stay with exchange, the 2007 version has some slick DR functionality. I deployed SCR (standby continuous replication) at our company earlier this year so that we have a live replica of the live DB at a standby site that can be activated at any point.

With purse strings so tight at present, work loads at bursting point and staff thinning not expanding, a move to 2007 in the near future is unlikely.

Maintaining cold-start systems is always tricky, and I'd probably push for an online DR solution instead. This also cuts the risk of hardware failing while it's not being used, and going un-noticed until the fateful day when it's actually required.

Our whole DR suit is online, about 20 odd cabinets worth of kit, it just so happens that the only two boxes offline are the Exchange ones - I agree about coldstart being a potential nightmare!