When will "penny-wise, pound-foolish" businesses ever learn? In this "true IT blooper," being a cheapskate about backup led to disaster for a financial services company.
Every story in our bloopers series comes to us directly from a TechTarget reader. In some cases, and for obvious reasons, some readers prefer to submit their tales anonymously. This author, David Levenson, was willing to go public. Here is his story.
Years ago, Levenson worked as an IT expert for a value-added reseller. The VAR offered several levels of disaster recovery support. Low-cost services, for example, included testing a customer's backup tapes once a month on the VAR's equipment. There were midrange offerings, as well as high-end solutions, such as creating an off-site recovery center to be used if the client's building was destroyed.
The VARs' standard practice was to recommend disaster recovery solutions that fit a customer's data protection and budget requirements. One of the VAR's clients, the aforementioned financial services firm, flatly refused to invest in any disaster recovery solution.
As a stopgap measure, the VAR's team created a checklist that would help the company test their backup tapes and determine the best ways to store them. "We also suggested that they purchase a second tape drive for testing their backups, as well as for redundancy if their unit failed," Levenson said. "They chose to ignore our advice." Instead, the company relied solely upon their minimalist backup reports, which only said if a backup had or had not been completed.
What happened next will come as no surprise. "Towards the end of the year, they called us in a panic," Levenson said. "They couldn't restore vital data from the beginning of that year."
The VAR's team analyzed the problem and discovered that that a tape drive's read-write heads had gone bad. "The unit thought it was writing to tape, but it wasn't," said Levenson. The standard backup reports used by the financial services firm didn't reveal this information. "They weren't able to recover their data. We couldn't restore their data using our tape drives either," he said.
After that, it was pretty easy to convince the customer to invest in the VAR's disaster recovery solution.
In terms of best IT practices, the moral of this story is that tape restores should be tested frequently. Be sure to test tapes before they are placed in long-term storage. "Also, try restoring from tapes using a different tape drive," Levenson advised. "You don't know if the tapes being written can only be read by a particular tape drive."
The moral for business bean counters in general is very simple: When it comes to disaster recovery, you can pay now or pay later.
Share your bloopers with us. E-mail them to firstname.lastname@example.org. Read more of our past IT Blooper Series, which originially appeared at SearchWin2000.com, part of the TechTarget network.