Exchange 2003 Fail Server

Holy hell, it’s hard to recover mail from an Exchange server if you’ve lost your domain controller. But I’ve done it. Once again, I’m a legend.

On Monday, a customer’s Windows 2003 Small Business Server went down due to a planned power outage. When it came back up again, the drives were corrupt. Their most recent backup was over a year old, and the RAID mirror had replicated the corruption.

I spent the better part of Monday and Tuesday fighting with the machine. Before I did anything, though, I imaged the data. I then replaced the existing drives with new 1TB drives to eliminate hard drive failure, and then began the arduous process of rebuilding data.

Active Directory was gone. I was able to log into the server using cached credentials. All attempts at either backing up or repairing the NTDS folder failed miserably. I reimaged the new drives probably four times.

Then on Wednesday morning, the server stopped booting. My boss and I decided to replace the hardware and begin rebuilding the server from scratch. This is an extremely painful process, because it means literally recreating years of configuration, certificates, licencing, e-mail and user data in a few days.

I was able to repair the Exchange database, using the built-in command-line tools. I made a backup of the EDB and STM files on a separate partition, along with the user shares and the Intranet website files (and SQL database), to prepare for the server rebuild.

Yesterday, Thursday, it became clear that Windows 2003 was not going to run on the new hardware we specified. What ever I tried, Active Directory kept failing. Whether at the initial AD setup phase, or during the Exchange 2003 installation, we got error after error.

So it was with a bold and insecure step that we decided to put our faith back in the original server. As the hard drives had already been replaced, I decided to replace the power supply as well.

As of right now, the server is still going strong. I believe that the power outage caused damage to the power supply in the server, which in turn caused corruption on the drives. With a new supply, the server has been performing admirably (despite being a Dell).

Tonight, I decided that I would spend some time this long weekend to recover the Exchange mailboxes. If you know anything about Exchange, you’ll know that it is deeply tied to Active Directory. Since I’ve had to rebuild the server (and change the domain name to avoid conflict when we reconnect the server to the network), all associations between the old Exchange database and the new domain are severed.

There are tools that Microsoft provides, to allow reconnection of the Exchange store. However, there are some steps one must follow:

1. Ensure that you’re logged in as a Domain Admin. It’s the right thing to do in this case.

2. Shut down the MSExchangeIS service and swap the EDB and STM files with the ones you’ve recovered from the old server.

3. Restart the Information Store service, and mount the store you’ve switched in.

4. Make sure the user has full rights on the Mail store, including Receive As and Send As permissions.

5. Now the fun part: if you’ve changed permissions, you should restart Windows. It’s quicker than waiting for the AD to replicate permissions (usually 15 minutes, but can take longer). I was fortunate in that I set up a LAN consisting of only one machine: the server.

6. Once in Exchange System Manager again, hook up the Administrative Group display. It gives you more options. You’ll have to go out and come back in again.

7. Now create some AD users. Make sure you do not give them email addresses, or associate them with Exchange. The reason is simple: you want to assign the old mailboxes with the new users.

8. In the mail store, set the Limits to 9999 days for keeping items. This will bring across every item associated with the old mailbox for the user.

9. In the Mailbox Recovery Center, hook up your mail store. You may need to refresh something, or run the Cleanup Agent or something. I forget now.

10. Right click on an account, and Find Match. If the AD user you’ve created has the same name, you’re in luck. You can then Reconnect it and it’ll be happy.

11. Once you’ve reconnected all of the mailboxes, start up Exmerge. This tool allows you to export and import mailboxes in PST format. Very nice. Very useful.

12. Using the two-step process, export all the mailboxes from the mail store to a folder on your hard drive. Make sure the errors are managed accordingly.

13. Shut down the Information Store again, and switch back the new files.

14. Restart the Information Store and mount the store if necessary.

15. Using Outlook from each users’ machines, import each PST file back into Exchange. Personally, I’d do this route instead of maintaining the old mailboxes, because you can take this opportunity to clean up each mailbox.

And the award goes to …

I have often imagined myself winning an Academy Award®. I have not decided what I would win it for yet, which leaves the probability more realistic than not. I even have my acceptance speech (and crowd-killing joke) already prepared.

But on a more serious note, I did a little bit of searching tonight on Google, for my paternal grandfather’s name, because that’s what I do whenever anyone mentions Dwight D. Eisenhower. Yes, that one.

After a bit of tweaking to tell Google to stop suggesting “Brian” instead of “Bryan” (after all, if you can’t spell your own grandfather’s name, after whom you are named, you have issues), I discovered that he received two fairly interesting decorations (amongst others of which I am no doubt unaware).

The first is a Bronze Star Medal, which is awarded by the US military for “bravery, acts of merit, or meritorious service”, according to the article on Wikipedia. It was awarded to him in 1945.

The second is Officer of the Order of Leopold, which is awarded by the Belgian King “for especially meritorious deeds”, according to the article on Wikipedia. It was awarded to him in 1948. Other recipients include Dwight D. Eisenhower and George S. Patton.

What is interesting is not that he received these decorations. My grandfather was, in a word, awesome. He became a Colonel in the British Army, and was with Eisenhower in Europe, including Belgium. What is interesting, then, is how humble he was about his actions in the war, and therefore how little I know about him from that time.

I only knew him for the first thirteen years of my life, when he was long retired and selling carpets, but I always liked him. He commanded respect. It’s hard to explain. I’ve noticed that about ex-military men. If he did ever speak about the war with me, the impression I got was that he was just one of many people doing their best to stop the Germans from taking over Europe. He didn’t gloat or brag. That made him even more awesome.

After I conducted a long interview with my maternal grandfather in 1991 about his own activities in the war (for a history project), he began telling me of my paternal grandfather’s achievements, which made me respect the man even more. How could I not?

I’m proud to carry his name (my second name is Bryan, in case you haven’t figured it out by now), and I can only hope that he’ll appreciate his special mention when I accept my Oscar.