Cuts in staffing have eliminated the one category of workers most IT departments need the most: the post-traumatic...
stress (PTS) counselor. At this very moment, systems are crashing and backups are being deleted, and system administrators around the world are banging their heads against walls. As PTS counselors know, the trauma doesn't end when the system glitch is fixed. For example, JL still gets the shakes when he remembers the day he dumped his domain.
JL's story started with the decision to install Novell 6.0 when it was new on the market. He figured that the Novell release, which was new at the time this story takes place, had security features needed by his network of 15 Windows NT-based servers.
JL is the only administrator for a large institution that has two sites connected via a WAN connection. He serves 600 users. Even though he is very proficient in NT and Novell, he decided to enlist help from an administrator who had completed many successful Novell installations.
Prior to the installation, JL built a lab to test the migration of NT to Novell. "I had a lot of trouble with it," he said. "Every time I tried to integrate the NT domain into Novell I would wipe out my domain."
Since he's a one-man IT shop, JL didn't have a chance to test the migration more than a few times. He never debugged it. So, he thought it best to leave it to the Novell migration expert. "You can see where this is going," JL said. In other words, start the "Jaws" theme music here.
Migrating from NT to Novell involved "simply installing the Novell client on the main domain controller and following a simple wizard with very few options," said JL. Before the Novell expert began the installation, JL said: "Are you sure this works? Every time I had tried it in the lab, it wiped out the domain, leaving the user unable to log on and removing all of the users and groups from NT. In effect, it destroyed the domain."
Both JL and the expert knew that destroying a domain was no laughing matter. "Rebuilding the network from scratch could take months if the backup didn't work," said JL.
The expert reassured JL that he had never had trouble with migrations of other versions of Novell. "He seemed confident, so I let him go," JL said.
The expert installed the Novell client, and the server didn't reboot. "Bad sign," said JL. Troubleshooting revealed that the onboard video card had failed. JL quickly installed a new one and up it went. "But the fun was just beginning," he said.
SAP via RSS - Subscribe to SearchSAP.com's RSS Feed for news and tips on SAP.
Having fixed the hardware problem, JL thought his problems were over. The duo proceeded to migrate the NT into Novell. The error messages that had popped up during JL's migration tests appeared. "This is when I started getting paler," JL recalled.
More troubleshooting ensued. They decided to back out of the installation, a tactic that the expert said "usually works." Just as he said those fateful words, he clicked "OK" without fixing the errors. The server promptly rebooted without notification and promptly wiped out the domain. No Users or Groups remained in the domain.
The help desk phone started to ring and continued to ring for hours. By this time, JL said, he was very, very pale.
"It was so bad I could no longer even log on locally to the server," said JL. "After much effort, I managed to get logged on locally to uninstall the client." The distraught duo also restored the SAM, which brought back the domain users and groups on the main controller. Even so, the domain on the backup controllers was destroyed.
By late that night, they had stabilized the system. Users could log on, but many had lost rights to certain shares. JL had to put that problem on the back burner. The backup controllers were still shaky, so JL built another quickly to stabilize the network. After that, "at least people could work," he said.
JL learned several important lessons from this disaster. "Never make changes that have far-reaching implications live on the network without testing thoroughly first," he said. Secondly, even experts make mistakes. "If you don't feel comfortable with the way things are going, stop and test it again," he advised.
The two tired administrators retried this procedure out in the lab the next day. After a lot more work, they discovered that there was an extra step in this installation that was not in the previous versions. "We got it to work perfectly in the lab and then did it live with no problems," he said.
Now, all is well with JL's network. As for JL, well, he's got a few more gray hairs and a minor case of PTS. That stress, however, is mitigated by his realization that "it could have been much worse!"
Share your bloopers with us. E-mail them to email@example.com. Read more of our past IT Blooper Series, which originially appeared at SearchWin2000.com, part of the TechTarget network.