A lovely day at work | Rickard Andersson

A lovely day at work

I had a wonderful day at work this Wednesday. I rebooted a server running under VMware ESX after installing some updates. After the reboot, the server for some reason decided it wanted to do some time travel. It reset its clock to February 14, 2008. Despite the odd time travel, this kind of issue usually resolves itself after a few minutes when the server synchronizes its clock in the domain hierarchy. However, the server in question was the root domain controller (PDC emulator) – the server that is at the top of the time synchronization scheme in the domain. So, in a matter of minutes, most of the other servers and as far as I know, virtually all clients in the domain (~450), followed suit and also set their clocks back to February. This is when things started going wrong. The phone started ringing and people were coming into my room asking me what was wrong with the network.

At this time, I was starting to panic a bit. So I logged into VMware VirtualCenter and looked at the clock on the ESX servers. Two of them were accurate, but the third was indeed set to February 14, 2008 and NTP wasn’t running. I tinkered a bit with this and managed to force it to the correct date. Due to a bug in ESX, I was however unable to enable NTP via the Infrastructure Client, but that’s another story.

Then something even more strange happened. All of a sudden, the top domain controller was convinced the date was September 29, 2009 (!). Before I noticed this, the date had begun replicating all over the network again and now things really started to break. People were unable to login, network printing failed, mapped network drives became unavailable, software licenses expired etc.

And now, the grand finale. As a security measure, we have developed a Windows service that runs on the network and disables user accounts in Active Directory that haven’t been used in 6 months or more. Guess what this service did when it thought the date was September 2009? Well, it disabled all user accounts in Active Directory of course. All 6500 of them! When I got the e-mail from the service, I really started to panic.

I immediately reset the clock on the domain controller and started working on a way to re-enable all the user accounts that had been disabled. This turned out to be a lot easier than expected so that particular problem was relatively easy to remedy.

Everything wasn’t nice and dandy though. Replication between the domain controllers was royally f*!#&d. Because the time had previously been set back to February 2008, the domain controllers believed replication had not taken place in about 10 months. This led to this lovely error:

It has been too long since this machine last replicated with the named source machine. The time between replications with this source has exceeded the tombstone lifetime.

The proposed solution is to demote the domain controller that has lost connection with the forest, remove inconsistent deleted objects with repadmin and then restart replication. I wish! For some reason, the domain controllers were refusing to talk to each other so we were unable to demote the controller that was out of whack. We had to forcibly seize the roles from the broken controller, remove it from the domain completely and then remove all traces (metadata) of the old controller on the functioning controller using ndsutil. After many hours of scratching our heads and with some help, we were successful. We ended up with only one domain controller, but that we can live with for a few days.

It’s on days like this you really feel that you earn your pay.

8 comments

  1. Theo
    Posted December 6, 2008 at 00:36 | Permalink

    I feel your pain mate! It’s times like this you want to disappear from mother earth. Not that this has happened to me but I don’t run domain controllers in a virtual environment and this won’t get me to adopt that strategy any sooner.

    Why the hell did the time warp forward in time? It could be useful to know for sure. It’s also worrying that just a date change can mess up replication beyond all recognition but I’m not surprised, not at all. I could almost figure that out early on in your post that this wasn’t going to be a good afternoon for you.

    It’s good that you can cope with just one domain controller because in a environment (not yours obviously) with several different sites it could have been worse off or even a forest with many domains. Don’t even want to think about it.

    This is not a scenario you could possibly even think or plan for. I’m happy to learn this from someone else’s experience and not first hand. :)

    This day another sad thing happened, Honda withdrew from Formula 1. Not a particular good day for Rickard.

    As fellow Mac-owner-F1-fan-Atheist-Toyota-owner-Windows-admin I find it funny how many things seem to have changed during the time we last spoke, many we now share. Last time we had quite identical computers from what I remember and pretty similar views on forum software development among others.

    We should hook up sometime which we never did in person. :)

    Peace,
    Theo alias Quizor

  2. Posted December 6, 2008 at 01:32 | Permalink

    Wow! Long time no talk :) We should definitely hook up in person some time. A Windows admin, huh?

    First of all, it is indeed a sad day for Formula 1. I just hope they can get back in the game in a few years when the economy has settled down. For next year, I’m just hoping Toro Rosso can keep improving the way they did at the end of this season. And that McLaren fail of course :)

    Regarding the time warp forward in time, I honestly have no idea. At first I thought it could have been me that accidentally set the year to 2009, but why September 29? It doesn’t make sense.

    I don’t think running a domain controller in a virtual machine is a bad idea in general. Running ESX servers with the clock off by 10 months on the other hand, that I do not recommend :)

  3. Theo
    Posted December 7, 2008 at 14:57 | Permalink

    Windows, sad isn’t it? But it pays my bills.

    My bet is that either Prodrive finds enough money to convince Honda to sell it cheep or that team is out in the cold as in the case with Super Aguri. I just don’t see how anyone would be able to find money out of the blue right now.

    This economic crisis probably means an end of an era. Rule changes are going to be adopted sooner. Few years from now they will be running standard engines. I’m happy that I went to see Formula 1 live in 2004 when they were still running proper engines, well just about everyone except McLaren.

    Looking forward to further reports on the domain controller failure if more information arrives.

  4. Posted December 16, 2008 at 00:24 | Permalink

    Sorry for the delayed response. I missed the e-mail from WordPress.

    While I agree that standard engines would be bad for the sport, I’m not all that worried. As far as I know, there aren’t any teams that are positive towards the implementation of standard engines. At least not among the big teams. I doubt it will be implemented anytime soon. Sure, for a couple of years, they’ll lower the rev limit and increase the required life expectancy of the engines, but I very much doubt they’ll go over to completely standardized engines. In 2010 or so, when the economy starts to bounce back, I think F1 will as well.

    The engines were amazing in 2004, but on the other hand, the racing wasn’t all that fun. Schumi won 13 of 18 races. The last two years, I think the racing aspect of F1 has improved dramatically, regardless of regulations. The engines might have 750 horsepower instead of 950, but the races have been more exciting to watch.

    I envy you for having seen Formula 1 in person. It’s something I have wanted to do for a couple of years now.

    Who do you think will take home the 2009 championship? It’s difficult to predict the performance of the different teams due to the big aerodynamics changes, but I think Alonso will be hard to beat next year. He’s arguably the best at working with the engineers to perfect the car and that’s something that will be vital next year.

    Cheers

  5. Theo
    Posted January 10, 2009 at 11:19 | Permalink

    There’s still good probability of getting standard engines. These could be made by the teams themselves or by a third party manufacturer. In fact I have nothing against that now that the engine development is frozen anyway. It’s just a big big waste of money now.

    Substantial rule changes so probably any of the more historically successful teams will take the title. I’m also afraid we won’t se as close championship as we’ve seen the last two years. KERS could introduce a gap between the teams again because a few will get it right. There’s also new ground to be made with the introduction of slick tires.

    Tell me if you want to go to a race, I’ll join you!

  6. Posted January 12, 2009 at 13:15 | Permalink

    http://www.formula1.com/news/headlines/2009/1/8824.html

    I think I can get used to the rear wing, but that new front wing isn’t pretty :)

    I’ve been thinking about going to Spa this year. Do you know any good resources for getting tickets?

  7. Theo
    Posted January 15, 2009 at 18:03 | Permalink

    I think I can get used to the front wing, rear wing is a different story. It feels misplaced. A wider rear wing with less angle, possibly with a single element would have made the cars a lot nicer to look at.

    Don’t remember where we bought ours. I’m sure there’s plenty. Usually it’s possible to buy directly from the track promoters (http://www.f1belgium.com/).

  8. Cushyprattle
    Posted January 8, 2012 at 03:04 | Permalink

    What the Hell was all; that?

Post a Comment

Comments are moderated. Your email is never published nor shared.