Friday, September 30, 2011

Outlook and Global Catalogs

We had an interesting issue recently where Windows clients at a branch office site were getting global catalog services from a domain controller in a remote site. In our environment, we're running Active Directory and at the moment are still at forest level 2003 R2, although over 90% of our DCs are Windows 2008 R2. We have close to 300 sites arranged in your traditional hub-and-spoke replication topology, and we've put two domain controllers at 80% of the sites to ensure that if one DC goes down there is still a DC/GC available locally.

Anyway, at this one site, the users sporadically went for GC services to a hub DC instead of one of their two local ones.

We checked all the usual suspects, including:
  • Was the client in a subnet that was inadvertently mapped to the wrong site, overlapped other mappings or not mapped at all in Active Directory Sites and Services? (That's the most common answer.)
  • Had an over-zealous admin hacked the registry and hardcoded the closest GC setting? Microsoftt KB 319206
  • Were over-zealous network admins blocking ports again? (GC services answer on ports 3268 and for secure LDAP, 3269)
All the normal answers were "no," and it was working most of the time, just not all the time.

So I asked the site to do a network trace from a client the next time they had an issue when they opened Outlook. Bingo. You gotta love traces.

Looking at the trace, it took all of one minute to find the problem. Another minute to fix it.

I filtered the trace looking for DNS queries and the first query I saw a DNS query for LDAP. The query was sent to the local DC.
Good so far.
Except that it returned information about the two local DC/GCs plus the remote DC/GC. hmmm

I opened DNS and drilled down to _tcp.Sitename._sites.gc._msdcs.parent.domain for the site having the issues. Sure enough, there was an _ldap record registered for the remote DC there. That's why, when clients asked for LDAP servers in the site, they had a 1 in 3 change of getting that remote DC.

So I deleted the _ldap._tcp.Sitename._sites.gc._msdcs.parent.domain record from DNS so the remote DC/GC would no longer be offered as a good alternative.

Why did the remote DC/GC get in there in the first place?

We have speculated that it's due to auto-site-coverage and that on 6/6/11 (the date when the remote DC created its _ldap record in that zone) the two DC/GCs in that specific site were having network issues or down or otherwise unable to cover their site. So the remote DC/GC stepped up to the plate and registered to cover the site since there were no other DC/GCs to cover it.

Please note that this is probably not a common issue, but it does happen. It's happened maybe 3 or 4 times to us in about 10 years.

Hope this helps someone....

Wednesday, April 6, 2011

Stuff I Should Know

This is more for home/small office folks who don't have separate IT staff. It's something I should have recognized much earlier in the game, but just didn't.

I've got quite a large network at home that required 8-10 CAT 5 cable connections and three wireless. Today, there are some wireless routers that incorporate 8 (possibly more) ports for wired connections, but I don't have one of those. I have the standard wireless router with 4 wired connection ports and had to supplement that with a second networking device (either another router or a switch--it doesn't matter, I've used both).

Now, being the idiot that I am, I just plugged everything up, did some elementary poking around the cute web pages to set the admin passwords to something private and set up WEP for security on the wireless. Then I thought I was done.

Silly me.

I forgot to look at the DHCP settings--you know, the service that hands out the IP addresses to the devices on your network so everything can talk to everything else. Actually, I did look at the DHCP and it looked good to me and I didn't think another thing of it.

All the devices got IP addresses. Everything could get to the Internet. I figured, well, good. Done.

Except, some devices could get to the wireless printer and others couldn't. My husband bugged the heck out of me, asking why he couldn't print. Some computers would show up in the HomeGroup and others wouldn't. If I shifted around the connections and wired up some of the computers to use the ports on the wireless router, they could use the wireless printer, but other devices on the network using the other switch (or router--my switch just burned up and I had to replace it temporarily with another router until my new switch arrives) couldn't print.

It was very aggravating, especially since I knew "the workaround" was to plug everything that needed access to the wireless devices to the ports on the wireless router. That should have clued me in sooner to the problem.

I looked again at DHCP, first on the switch and then on the wireless router.
The switch was using 192.168.0.1 to 192.168.0.254 with a mask of 255.255.255.0
The wireless router was using 192.168.1.1 to 192.168.1.254 with a mask of 255.255.0.0.

So the devices getting IP addresses in the 192.168.0.0 network were essentially under the impression that devices in the 192.168.1.0 were in a different network. An unroutable network.

I changed the switch's IP to 192.168.1.x and corrected the mask, and then I split up the IPs to give DHCP on the switch the lower range to hand out while the wireless router got to keep the upper range of IP addresses to hand out so there would be no IP address conflicts.

Now, we can all print and we all show up in the HomeGroup network.
This is...like...networking 101 but it shows how easy it is to overlook the basics.

Tuesday, May 18, 2010

Violate Best Practices at your own risk

Learned something new in one of those crazy, all night "the sky is falling" scenarios that turn out to be a stubborn admin shooting himself in the foot. He chose to ignore best practices that state: put users into global groups, and then global groups into...well...whatever is necessary.

He decided to put users into universal groups in another domain, even though all the users were from one domain and should have simply been put into a global group in their own domain.  The reason? Politics and stubbornness.

You see, we were forced to create some additional domains against our best advice. And then, again, despite our advice to just put centralized computer resources in those domains (a la the old NT resource domain model) they decided to put security groups in that domain, as well. Because they wanted to.

That made the groups Universal.

On the surface, that seems fine...except for the following:
Scenario Setup
1)  All the users were in domain X
2) The computers were in domain Y
3) The security groups were in domain Y (for no reason except political)
4) The OUs containing the users in domain X are being renamed and reorganized
5) All the DCs in domain X and domain Y are GCs, except two per domain: 1 for backups (DIT is smaller); and 1 for the Inf Master.
6) All the DCs that are GCs got the message about the OU changes.
7) But...there are phantom records in the Universal groups that only get updated about every two days...so...the groups broke because for some inexplicable reason, the servers consuming the groups fell in love with the Inf Master role holder.

Ouch.

Phantom Records
Most of us tend to forget about those lovely little phantom records. See Microsoft KB 248047 for details. Basically, non-GC DCs have no direct knolwedge of objects in other trusted domains, because they don't have a GC. So groups like universal groups containing accounts from trusted domains in Active Directory insert a phantom object for these cross-domain group-to-user references.

Phantom records only contain:
  • The distinguished name of the object
  • The GUID of the object
  • The SID of the object
When a user is added to a universal group, for example, in another domain, any non-GC DCs add one of these phantom objects representing the remote user. That way, they can "intellegently" display and identify the user when someon peaks into the group, without requiring access to a GC.

So when these folks renamed the OUs containing the users, the distinguished name for the user object changed. And access to the systems using those security groups containing the bad user DNs stopped.

And we had an all night call. People tried to force replication. They tried creating a new group and adding the users--but low and behold--the users added to the new group showed the same bad DNs on the non-GC DCs!

It had nothing to do with replication. Replication was working fine. Those phantom records were to blame.  Well, no. Not true. The real cause was the decision to put groups, which would normally be global groups in domain X (per best practice), in domain Y. This made them universal and yada, yada, yada.

You see, non-GCs only rescan their phantom objects and refind/correct DNs about every two days. Because it is a "labor" intensive operation. If it did it more frequently, it might not even finish before it had to start the process again. And then there's all that nasty WAN traffic and DC non-responsiveness while it chugs through this process.

Now you can kick off a phantom scan manually (which we did) but you don't want to do so lightly. And an hour after I kicked off the phantom scan, someone actually renamed the OUs again, anyway, thereby causing the problem all over again. Nice. Thanks.

The interesting thing to me is that the folks managing this do not want to correct it by simply making global groups in the appropriate domain. Instead, they wanted directions on kicking off phantom scans, themselves, whenever they felt it necessary. Like every five minutes or so.

Let's just say that's why they are not Enterprise Admins.

So, sorry. If you shoot yourself in the foot, you have to expect to bleed. And don't expect me to load your gun for you.

Wednesday, March 10, 2010

VBA Macro for Outlook to Save E-mails to File

This is hardly stupendous, and I found various versions of this on the Internet, but sadly, most of the versions simply did not work. The following was my final "massaged" version that actually worked pretty well. Not perfect, but it worked.

TASK
The task was to write a macro for Outlook that would save the e-mail as a straight TXT file, using the subject as the filename.

For example, if I got an email with the subject: This is my agenda
It would save that e-mail in a file as: This is my agenda.txt

There are some characters, however, that people insert into subject lines that prevent the file from being saved. For example: \, /, *, |

So I had to strip those out before saving the file.

Here is the final macro:

Sub save_email(myItem As Outlook.MailItem)
   Dim tempstr As String
   Dim strname As String
   Dim Badchar As String
   Badchar = "\/:*?<>()"  
   Dim intIndex As Integer
'pick up the e-mail subject

   tempstr = myItem.Subject
'Remove special chars

   For i = 1 To Len(tempstr)
     If InStr(1, Badchar, Mid(tempstr, i, 1), vbTextCompare) Then
        strname = strname & ""
     Else
        strname = strname & Mid(tempstr, i, 1)
     End If
   Next
'Save to a file

myItem.SaveAs "c:\Users\tuser\My Documents\" & strname & ".txt", olTXT
End Sub

----------
That's it. The piece that was missing from most of the things I've seen posted on the Internet was the strname = strname & "" line where it actually removes the bad char and keeps on building strname with the characters from the e-mail's subject. I have no idea why other people's programs worked when you only specified strname = strname  but the script failed to remove the special characters when I did that, so I had to add the & "" part which essentially adds nothing to the string.

Go figure.
At least it seems to work.

Friday, February 26, 2010

DNS, DNSCMD and Junk like that

I like to use DNSCMD to slam in a lot of DNS records (in bulk) when we're tearing down or moving zones around. I generally use a batch file wrapper around it, to parse a text file with my DNS record information, like host names and IPs.

Today, I had the challenge of taking a little over a hundred host, MX, and CName records and shoving them into a new setup. And I had the devil of a time getting the syntax right.

Looked all over the web and really didn't find that much info that was helpful to me, particularly with the CName and MX records. Loads out there about Host "A" records, though.
Here is Microsoft's idea of help with DNSCMD:

dnscmd [ServerName] /recordadd ZoneName NodeName RRType RRData


Adding MX Records with DNSCMD
This was much trickier than I expected. I ended up with two forms of the command.
Form 1 (Same as Parent)
You use this when the the name for the mail handler is the same as the zone name.  If you look in your DNS console, you'd see this represented as:
DNS Zone = test.org
(same as parent)          Mail Exchanger (MX)     [10] mymailserver.test.org

So you want an MX record for test.org that sends traffic to mymailserver.test.org

The DNSCMD to do this would be:
DNSCMD mydnsserver.test.org /RecordAdd test.org @ MX 10 mymailserver.test.org

The command is color coded so you can see how it all lines up with the information in DNS.

Form 2
You use this form when the name for the mail handler is different from the zone name. If you look in your DNS console, you'd see this represented as:
DNS Zone = test.org
hq                              Mail Exchanger (MX)    [10] myhqmailserver.test.org


You want mail sent to hq.test.org to be handled by myhqmailserver.test.org

The DNSCMD to do this would be:
DNSCMD mydnsserver.test.org /RecordAdd test.org hq MX 10 mymailserver.test.org

Again, it's color coded so you can see how the elements align. Assuming you can see the colors. :-)

Adding a CName record
In some ways, this is easier to understand than the MX. You have an alias for another server that you want to poke into DNS.
Again, you have your three elements, those being: the zone where your server lives, the name of the server, and the alias/cname you want to assign it.

Let's say you have sql1.test.org and want to give it a cname of bigserver.test.org, here is the DNSCMD to do that:
dnscmd mydnsserver.test.org /RecordAdd test.org sql1 CNAME bigserver.test.org

It's color coded, too.
Hope this made some sort of sense.
That's it for now. Have a great evening.

Tuesday, September 29, 2009

RPC Server Unavailable and DNS

Resolving One Weird Case of RPC Server Unavailable

I've been battling other problems (like lingering objects when we've never done a restore from a backup tape or had any of our DCs offline for more than a couple of days) when I ran across one of those annoying "RPC Server Unavailable" messages from our replication monitoring.

I was already working with Microsoft on the whole "where are these lingering objects coming from" thing, so I dragged them into the muck with me.

I thought our DNS architecture was beautiful and bullet-proof. I was wrong.
Note: I've very much simplified our "architecture" in this example. In reality, we have 36 domains and around 582 domain controllers. (You don't want to know the real and extremely complex design.)

Symptoms
So we got the message "RPC Server unavailable" when DC2 tried to suck data from DC3. DC3 could suck data just fine from DC2.

Both DC2 and DC3 are members of an AD domain called weird.test.org. The DNS zone for weird.test.org is AD-integrated. There is one tree in the forest of test.org. All the DNS zones, in fact, are AD-integrated, and all the DCs also run DNS. All DNS servers are DCs.

DC1 is in test.org, the root domain of the AD tree. DC1 carries the test.org DNS zone and the DNS delegation for the weird.test.org DNS zone, which is delegated down to DC2 and DC3.

So you have:
DC1.test.org, carries the test.org DNS zone and delegation for weird.test.org
  • The _msdcs.test.org DNS zone also lives on DC1.
  • DC1 points to itself for DNS resolution.
  • DNS is configured to forward to some Internet gateway DNS servers for any zones it does not carry.
  • weird.test.org is delegated to DC2.weird.test.org and DC3.weird.test.org.
DC2.weird.test.org, carries weird.test.org DNS zone, and also carries the root domain zone: test.org
  • DC2 points to DC1 for DNS resolution.
  • DNS is configured to forward to DC1 for any zones it does not carry.
DC3.weird.test.org, , carries weird.test.org DNS zone, and also carries the root domain zone: test.org
  • DC3 points to DC2 for DNS resolution.
  • DNS is configured to forward to DC2 for any zones it does not carry.

At one point, DC3 had been on old hardware in a different site. We demoted and booted out the old DC3 and brought up a new DC3 on new hardware with a different IP address. This was several months before this incident.

Testing
When we pinged DC2 from DC1, it pinged fine using the fully qualified domain name, e.g. DC2.weird.test.org. But when we pinged the GUID _msdcs record, e.g. abe33cdf-edd2-3456-abc1-acbd12398745._msdcs.test.org, we got a the wrong IP. WTF?

Troubleshooting
I went through every DNS server, every DNS server on the delegation for DC2's zone, weird.test.org, and checked the host A records for DC2 on every DNS server with a copy of the zone.

All the DNS servers had the right data. The Host A records were correct. The PTR were correct. There were no duplicate Host A records. There were no duplicate PTR.

We cleared all caches (cleared the DNS server cache and did an ipconfig /flushDNS).

We did a network trace to track who was the culprit with the bad IP information for DC3.weird.test.org. The trace showed that the name resolution process never left DC1. Even though DC1 was not authoritative for the weird.test.org zone and therefore did not have the DC3.weird.test.org Host A record to be able to come up with an IP for DC3.

Weird Discovery

Because DC2 uses DC1 for DNS resolution, it asked DC1 for resolution of abe33cdf-edd2-3456-abc1-acbd12398745._msdcs.test.org.

The _msdcs.test.org DNS zone contains cname records for the domain controllers (DCs), so abe33cdf-edd2-3456-abc1-acbd12398745._msdcs.test.org is a CNAME that resolves to: DC3.weird.test.org.

DC1 therefore looked for a delegation to weird.test.org.
DC1 found the delegation to weird.test.org. But it also found the name it was looking for on the delegation.

"A-ha!" DC1 said. "I don't need to actually query any of the authoritative DNS servers listed on the delegation. I have the IP right here on the delegation itself."

So DC1 returned the IP listed on the delegation rather than asking DC2 or DC3 (the two DCs authoritative for the zone).

And unfortunately, because we had replaced DC3 and changed the IP at that time, this information on the delegation never got updated.

Moral of the Story
It should have occurred to me that if DC1 needs to resolve the IP for a DC/DNS server listed on a delegation, it will just use the IP on the delegation and not actually send a request for resolution to the DC/DNS servers authoritative for the zone. It makes sense that it does that--why forward a query to the DC when you already have the IP? It was just unexpected.

I never thought to look at the delegation to find the bad IP DC1 was handing out.

And it also embarrassed me, because I thought we had corrected that IP when we replaced DC3.

Would-a; could-a; should-a.
Oh, well. One more learning experience.

Tuesday, September 1, 2009

Tricked-out eeePC




Tricked-out eeePC 901

Insane or not, I recently spent a fun-filled weekend maximizing the potential of my ASUS eeePC 901--or netbook--if you like that term. What did I do?
First, I bought some really cool skins (aka decals) as you see. They seemed so apropos since in my admittedly scarce spare time, I also write mysteries. My eeePC is resting on top of my "normal" Dell laptop so you can see the size comparison.
Well, let's see, what else did I do:

Hardware Upgrades
  • Memory upgrade: sTs Electronics ASUS eeePC 901 2GB RAM memory upgrade replaced the whimpy 1GB the system came with.
  • Solid-State Drive Upgrade: 32GB SaberTooth-SS SATA Mini PCIe SSD for 901 eeePC upgrade to replace the tiny 16GB SSD installed in the system (the system says it has 20GB, but it's divided between a 4GB and 16GB SSD, so I replaced the 16GB with the 32GB). I would have gone for 64GB, but it was beyond my means at the moment.

  • Additional Storage: Transcend 16GB SDHC Class 6 Flash Memory card with card Reader. (I had evil designs for this extra bit of storage).
  • Additional Storage #2: I already had an extra 1GB USB stick lying around...
Method (to my madness)
Once I got all my bits and pieces (or is that bytes and pieces? ;-) ) I...
  1. Updated the BIOS--this is a really important step because the system wouldn't recognize the new SSD until I did this.
  2. Installed all the new hardware.
  3. Followed the directions at the blog Install Windows XP on the Asus EEEPC to install Windows XP on the 1GB USB stick and make it bootable from my main desktop PC.
  4. Turned on the system and went into the BIOS (press F2 as the system boots) and told it to boot to the 1GB USB stick.
  5. Booted to the 1GB USB stick and initiated the installation of Windows XP
  6. Installed XP on the 4GB partition, overlaying the old OS
  7. Created a D: drive out of the 32GB SSD and formatted it for NTFS
  8. Copied the source code for Windows 7 to the Transcend 16GB Flash memory card (formatted for NTFS) from my desktop PC.
  9. Stuck the 16GB Flash memory card into the eeePC and initiated the Windows 7 install.
  10. Installed Windows 7 to the D: drive, leaving XP on the 4GB partition (C:) as a failsafe in case I ever get a bunch of money and can replace the 32GB SSD with a 64GB SSD :-). This makes it easier for future upgrades.
  11. Whoopee! I now have Windows 7 (with Aero glass) running on my eeePC and I'm happy as a clam.


Windows 7 Running on an eeePC 901


And the lovely, Aero-Glass Windows 7 desktop. Yep. It works.