AD DHCP-DNS troubleshooting

[Little_Crow]

Got a very strange issue with our DHCP service that started yesterday (Nothing changed on the server that I'm aware of), we have pretty much all of the leases with the pen\brush icon indicating DNS has not been updated.

If we restart the service, DNS registration will work fine for 20 minutes or so, then it will just start failing again.

I've checked the DHCP log files and these indicate a failure, but no more information than that. Event logs on both the DHCP and DNS servers indicate no failures.

I've switched on debug logging on the DNS servers but I don't even see requests going across, suggesting the DHCP server isn't even sending the updates.

I've googled everything I can think of, but can't find anything relevant, my next step is to install wireshark and packet sniff it, unless someone here can give me a magic answer on this.

Help me OCUK, you're my only hope.

#########
Additional info:
It's 1 DHCP server and 2 DNS servers, all W2k3 DC's

 

[Little_Crow]

Some further info from this morning troubleshooting:

Checking the DHCP logs further it suggests that our DNS server is taking 3+ minutes to register the workstation before the "DNS Update Successful" message is logged in DHCP.

Since I've never needed to look at this before, could someone advise me/check their own logs to tell me how long this should normally take?

I think this may be a symptom of our DNS servers being overloaded, anyone know how to further check this theory?

thanks,

Little_Crow

###########
Just had a renew request that was still waiting for DNS after 20 minutes

 

[Little_Crow]

You'd think that, but there really are no error messages in the event logs regarding DNS.

In the mean time, I've changed the adapter settings on the DHCP server to use a different DNS server first and rebooted, it's been running 45 minutes now, and DNS registrations are now happening in a timely manner.

So far it seems to support my theory of an overloaded DNS server, and our firewall logs show an enormous amount of external lookups coming from that DNS box too.

I'm going to see how things go over the next 24 hours and I hope to see the dreaded brush/pen icon disappearing. If all is well, i'll be looking into building a server dedicated to external lookups.

 

[Little_Crow]

erm... you should have the internal DNS server forward external lookups to your ISP.

This is already configured, I believe that most of the requests being funneled through it are from our internet proxy server (no direct external connection on it), my plan is to create a new DNS server to service these requests, so that internal queries are not being impacted by the external requests.

 

[Little_Crow]

Sounds like an odd way to do it.

If it was me I would forward any external queries from the proxy directly to the ISP forwarder, and not my internal DNS...

But there we go!

That does make a lot more sense to be honest.

Sadly though, the DHCP-DNS registration has collapsed again after working flawlessly for 45 minutes or so, back to the drawing board again.
Not seen any successful lines in the DHCP log, it's a sea of 'DNS Update Failed'

It looks like it started failing after the DHCP database cleanup began, not good but at least I have something to work from.

 

[Little_Crow]

We have rather a lot of scopes, so I'd rather not set them all up again.
At the same time I don't want to use the existing DB and restore it elsewhere as this may be where the problem lies.

I'm sorting out a restore of the DHCP DB from before the trouble started, I'm hoping this will fix this strange issue.

 

[Little_Crow]

OK, I'm now at a point where I think it really is fixed.

I've found pretty much a carbon copy of the issue I *was* having here.
The issue kicks in after the database cleanup runs, and has happened out of the blue despite working for years without issue.

It looks like it was a missing reverse lookup zone, I've added the zone into DNS, and very shortly after that the DHCP cleanup ran again. This time it hasn't been followed by a slew of failed messages.

Fingers crossed that it stays working. Thanks for the support, and your suggestions have also thrown up a couple of other issues to address.

 

[Little_Crow]

Strange, I wonder why the reverse lookup zone was missing!!

Not so much missing, as never having been created. It isn't a new subnet, so it's odd that it has been working for so long, only for it to suddenly break, but this is exactly the same circumstances as the thread I found on the technet forums.
2 or 3 cleanups have run on the server now, and there have been no failures at all, so it all looks like it's working.

Plans are afoot for the proxy servers DNS to be changed. Our pac file should already be directing any internal services accordingly before they hit the proxy, but I never underestimate the ingenuity of our users to find strange ways to access systems.