Tuesday, September 29, 2009

RPC Server Unavailable and DNS

Resolving One Weird Case of RPC Server Unavailable

I've been battling other problems (like lingering objects when we've never done a restore from a backup tape or had any of our DCs offline for more than a couple of days) when I ran across one of those annoying "RPC Server Unavailable" messages from our replication monitoring.

I was already working with Microsoft on the whole "where are these lingering objects coming from" thing, so I dragged them into the muck with me.

I thought our DNS architecture was beautiful and bullet-proof. I was wrong.
Note: I've very much simplified our "architecture" in this example. In reality, we have 36 domains and around 582 domain controllers. (You don't want to know the real and extremely complex design.)

So we got the message "RPC Server unavailable" when DC2 tried to suck data from DC3. DC3 could suck data just fine from DC2.

Both DC2 and DC3 are members of an AD domain called weird.test.org. The DNS zone for weird.test.org is AD-integrated. There is one tree in the forest of test.org. All the DNS zones, in fact, are AD-integrated, and all the DCs also run DNS. All DNS servers are DCs.

DC1 is in test.org, the root domain of the AD tree. DC1 carries the test.org DNS zone and the DNS delegation for the weird.test.org DNS zone, which is delegated down to DC2 and DC3.

So you have:
DC1.test.org, carries the test.org DNS zone and delegation for weird.test.org
  • The _msdcs.test.org DNS zone also lives on DC1.
  • DC1 points to itself for DNS resolution.
  • DNS is configured to forward to some Internet gateway DNS servers for any zones it does not carry.
  • weird.test.org is delegated to DC2.weird.test.org and DC3.weird.test.org.
DC2.weird.test.org, carries weird.test.org DNS zone, and also carries the root domain zone: test.org
  • DC2 points to DC1 for DNS resolution.
  • DNS is configured to forward to DC1 for any zones it does not carry.
DC3.weird.test.org, , carries weird.test.org DNS zone, and also carries the root domain zone: test.org
  • DC3 points to DC2 for DNS resolution.
  • DNS is configured to forward to DC2 for any zones it does not carry.

At one point, DC3 had been on old hardware in a different site. We demoted and booted out the old DC3 and brought up a new DC3 on new hardware with a different IP address. This was several months before this incident.

When we pinged DC2 from DC1, it pinged fine using the fully qualified domain name, e.g. DC2.weird.test.org. But when we pinged the GUID _msdcs record, e.g. abe33cdf-edd2-3456-abc1-acbd12398745._msdcs.test.org, we got a the wrong IP. WTF?

I went through every DNS server, every DNS server on the delegation for DC2's zone, weird.test.org, and checked the host A records for DC2 on every DNS server with a copy of the zone.

All the DNS servers had the right data. The Host A records were correct. The PTR were correct. There were no duplicate Host A records. There were no duplicate PTR.

We cleared all caches (cleared the DNS server cache and did an ipconfig /flushDNS).

We did a network trace to track who was the culprit with the bad IP information for DC3.weird.test.org. The trace showed that the name resolution process never left DC1. Even though DC1 was not authoritative for the weird.test.org zone and therefore did not have the DC3.weird.test.org Host A record to be able to come up with an IP for DC3.

Weird Discovery

Because DC2 uses DC1 for DNS resolution, it asked DC1 for resolution of abe33cdf-edd2-3456-abc1-acbd12398745._msdcs.test.org.

The _msdcs.test.org DNS zone contains cname records for the domain controllers (DCs), so abe33cdf-edd2-3456-abc1-acbd12398745._msdcs.test.org is a CNAME that resolves to: DC3.weird.test.org.

DC1 therefore looked for a delegation to weird.test.org.
DC1 found the delegation to weird.test.org. But it also found the name it was looking for on the delegation.

"A-ha!" DC1 said. "I don't need to actually query any of the authoritative DNS servers listed on the delegation. I have the IP right here on the delegation itself."

So DC1 returned the IP listed on the delegation rather than asking DC2 or DC3 (the two DCs authoritative for the zone).

And unfortunately, because we had replaced DC3 and changed the IP at that time, this information on the delegation never got updated.

Moral of the Story
It should have occurred to me that if DC1 needs to resolve the IP for a DC/DNS server listed on a delegation, it will just use the IP on the delegation and not actually send a request for resolution to the DC/DNS servers authoritative for the zone. It makes sense that it does that--why forward a query to the DC when you already have the IP? It was just unexpected.

I never thought to look at the delegation to find the bad IP DC1 was handing out.

And it also embarrassed me, because I thought we had corrected that IP when we replaced DC3.

Would-a; could-a; should-a.
Oh, well. One more learning experience.

Tuesday, September 1, 2009

Tricked-out eeePC

Tricked-out eeePC 901

Insane or not, I recently spent a fun-filled weekend maximizing the potential of my ASUS eeePC 901--or netbook--if you like that term. What did I do?
First, I bought some really cool skins (aka decals) as you see. They seemed so apropos since in my admittedly scarce spare time, I also write mysteries. My eeePC is resting on top of my "normal" Dell laptop so you can see the size comparison.
Well, let's see, what else did I do:

Hardware Upgrades
  • Memory upgrade: sTs Electronics ASUS eeePC 901 2GB RAM memory upgrade replaced the whimpy 1GB the system came with.
  • Solid-State Drive Upgrade: 32GB SaberTooth-SS SATA Mini PCIe SSD for 901 eeePC upgrade to replace the tiny 16GB SSD installed in the system (the system says it has 20GB, but it's divided between a 4GB and 16GB SSD, so I replaced the 16GB with the 32GB). I would have gone for 64GB, but it was beyond my means at the moment.

  • Additional Storage: Transcend 16GB SDHC Class 6 Flash Memory card with card Reader. (I had evil designs for this extra bit of storage).
  • Additional Storage #2: I already had an extra 1GB USB stick lying around...
Method (to my madness)
Once I got all my bits and pieces (or is that bytes and pieces? ;-) ) I...
  1. Updated the BIOS--this is a really important step because the system wouldn't recognize the new SSD until I did this.
  2. Installed all the new hardware.
  3. Followed the directions at the blog Install Windows XP on the Asus EEEPC to install Windows XP on the 1GB USB stick and make it bootable from my main desktop PC.
  4. Turned on the system and went into the BIOS (press F2 as the system boots) and told it to boot to the 1GB USB stick.
  5. Booted to the 1GB USB stick and initiated the installation of Windows XP
  6. Installed XP on the 4GB partition, overlaying the old OS
  7. Created a D: drive out of the 32GB SSD and formatted it for NTFS
  8. Copied the source code for Windows 7 to the Transcend 16GB Flash memory card (formatted for NTFS) from my desktop PC.
  9. Stuck the 16GB Flash memory card into the eeePC and initiated the Windows 7 install.
  10. Installed Windows 7 to the D: drive, leaving XP on the 4GB partition (C:) as a failsafe in case I ever get a bunch of money and can replace the 32GB SSD with a 64GB SSD :-). This makes it easier for future upgrades.
  11. Whoopee! I now have Windows 7 (with Aero glass) running on my eeePC and I'm happy as a clam.

Windows 7 Running on an eeePC 901

And the lovely, Aero-Glass Windows 7 desktop. Yep. It works.