Demoted DC with lingering DNS records, causing domain join errors
A story of pain and stupids
Encountered an issue, wasn’t able to find any clues on the web, so writing this to maybe help the next person.
Apologies for the sanitised screenshots, done for the obvious reasons.
In hindsight, the clues were pretty obvious, should have got there a lot faster.
Problem
Intermittent AD domain join tasks fail due to attempting to join against a DC that does not exist.
On the client side, might see an error like this.
The following error occurred when DNS was queried or the service location (SRV) resource record used to locate an Active Directory Domain Controller (AD DC) for domain ...
The error was: "DNS server failure."
(error code 0x0000232A RCODE_SERVER_FAILURE)
The query was for the SRV record for _ldap._tcp.dc._msdcs.YourDomain.Name
Cause
DNS records for old DCs still exist, these are supposed to be automatically removed when a DC is demoted right.
So maybe a little hard to tell from the screenshot, but there were DCs referenced here that had since been demoted, and there are more _ldap records than there are _kerberose
Checked Sites and Services, nothing there that shouldn’t be.
Just delete them?
Easy! Just delete them. Job done.
They look like they delete just fine, refresh the console, and there they are again.
Time to bring out the big guns; AD metadata cleanup
If you’ve ever had clean up a DC decommission gone bad (either your own doing / that last guy), you’ll be pretty familiar with AD metadata clean up.
https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup
LDAP error 0x22(34 (Invalid DN Syntax).
Ldap extended error message is 0000208F: NameErr: DSID-0310022D, problem 2006 (BAD_NAME), data 8350, best match of:
'CN=Ntds Settings,XXX'
Win32 error returned is 0x208f(The object name has bad syntax.)
)
Unable to determine the domain hosted by the Active Directory Domain Controller (5) Please use the connection menu to specify it
Well crap, I was feeling pretty confident about that.
Rabbit holes
Here is were I burnt a bunch of time going down a few rabbit holes thinking the records were being auto recreated.
Fixed a few other issues along the way that turned out to be unrelated, so maybe something just slightly less than a 100% waste of time.
Zone Name Servers
Clean up those name servers for each zone, forward and reverse.
Authorized DHCP servers
https://devblogs.microsoft.com/scripting/use-powershell-functions-to-authorize-dhcp-servers/
Found some old DCs were still marked as Authoized DHCP servers. Most deauthoized fine, some threw a “There is no such object on this server” error.
Can clean those up in ADSI edit under the “Configuratioin” naming context.
CN=NetServices,CN=Services,CN=Configuration,DC=xxx,DC=yyy
And bunch of other poking around to no avail.
Interlude
https://www.google.com/search?q=how+distinguish+between+impostor+syndrome+and+incompetence
https://www.google.com/search?q=how+to+computer
duh. Delete record by some others means… PowerShell?
Delete those damn DNS records by some other means, like PowerShell… why did I not think of this before?
And someone at Microsoft already had it worked out.
https://devblogs.microsoft.com/scripting/clean-up-domain-controller-dns-records-with-powershell/
“ObjectNotFound” … you just had it 2 miliseconds ago.
Ok so the DNS GUI console thing was silently failing, least now we’re getting somewhere.
Active Directory is DNS integrated LDAP (and Kerberos) … so must be an AD object class thingie, right?
These DNS records must be stored in AD somewhere, can we do the needful there?
https://docs.microsoft.com/en-us/windows/win32/adschema/c-dnsnode
Can’t find any matches for “_ldap._tcp.dc._msdcs”
But can find a bunch that match “_ldap._tcp.”
Ok, cool. Looks like they live under here somewhere
CN=MicrosoftDNS,DC=DomainDnsZones,DC=xxx,DC=yyy
CN=MicrosoftDNS,DC=ForestDnsZones,DC=xxx,DC=yyy
BAM!
Found it.
https://docs.microsoft.com/en-us/sysinternals/downloads/adexplorer
And it’s a byte array…
No idea why it didn’t show up in the previous search, but moving on…
Now to parse this thing
https://github.com/dirkjanm/krbrelayx/blob/master/dnstool.py
Had some help here from a friend and colleague to parse this thing, you know who you are. Thank you!
Using AD Explorer, copy and paste the lines from the dnsRecord attribute, paste into the following script.
Set-StrictMode -Version 'latest'
$ErrorActionPreference = 'stop'
$lines = @'
38 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
38 0 33 0 5 240 0 0 22 245 107 .............................................
38 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
38 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
41 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
41 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
38 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
39 0 33 0 5 240 0 0 22 245 107 .............................................
'@
$dtSummary = [System.Data.DataTable]::new()
[void]$dtSummary.Columns.Add('line', [int])
[void]$dtSummary.Columns.Add('dnsRecord', [string])
[void]$dtSummary.Columns.Add('bytes', [string])
$lineCount = 0
foreach ($line in ($lines -split [System.Environment]::NewLine) ) {
$lineCount++
$bytes = $line.Trim().Split(' ')
$Asc = ''
for ( $x = 24; $x -lt $bytes.Count; $x++ ) {
if ($bytes[$x] -eq 133) {
$Asc += ". "
} ElseIf ([int]$bytes[$x] -lt 32) {
$Asc += ". "
} else {
$Asc += "$([char][byte]$bytes[$x]) "
}
}
$row = $dtSummary.NewRow()
$row.line = $lineCount
$row.dnsRecord = $Asc -replace ('\s', $null)
$row.bytes = $line
$dtSummary.Rows.Add($Row)
}
$dtSummary | Sort-Object -Property ('line') | Format-Table -AutoSize
In AD Explorer, right click on the dnsRecord attribute, Modify…
Now we can correlate and delete based off the line numbers.
Refresh the DNS console to confirm that the expected happened … no change.
Close DNS, open again, check, ok good.
And domain joins work 100% every time, nice and boring like. Yay, done.
Very nice indeed.