Sunday, 9 November 2014

"Backups, why do I need to do backups?"

One thing that I find in common when I speak to people about problems that they are having with either their desktop or laptop, is that most people dismiss backups as an irritant that they can do without. They have never lost any data in the past so what will change in the future?

Having had some success in persuading people that they do need to do backups in one form or another (usually a failed or very poorly hard drive focuses the mind) I still find that it is an uphill struggle. University students are usually the best recipient to my mantra of "there are only three things to remember 1. Do your backups 2. Do your backups and 3. Do your backups. I usually also tell them the story of the third year medical student with a very poorly hard drive in their laptop and no backups, nothing - zero - zilch - nada. If it wouldn't come off that failing hard drive it was gone forever. Focus the mind, do you really want to lose all that work?

So why all the ramblings about backups?

Well I received a call from home this week asking why there was a funny smell in the garage? No idea was my reply. Then our son rings home and asks if the internet was off as he could not pick up his email. Aha light bulb moment. Son gets wife to check email server for power lights, there were none. "Pull the power lead out the back Mum, NOW!"

So what did I find when I got home? The email server is running on a HP ML110 G5 server with two 250GB SATA hard drives in Raid 10. One of the hard drives failed around 12 months away and was replaced and the raid rebuilt from the remaining good hard drive. When the side of the server was removed it was quite clear what had happened.

The lower of the two hard drives had suffered a melt down of the power connector.

When moving the power connector the power leads fell out of what was left of the melted connector. There was no insulation left on the cables and the copper conductors had fused together.

This is what was left of the power connector, just a big blob of plastic. You will see that I had plastic gloves on, mainly because of the soot but also because I was not sure if the melted plastic would be dangerous, some plastics can be toxic when burnt.

This was the upper hard drive. The power connector and the sata cable had melted but luckily the hard drive only suffered from superficial smoke damage. I did wonder if the hard drive had been heated to such a point that the solder would have melted.

I think that there were actual flames inside the server due to the amount of smoke and soot evidence. 

A close up of the memory modules shows the amount of soot that had been deposited. 

The soot being mainly carbon and therefore conductive, I thought it was wise to clean off as much as possible. With the careful use of some isopropyl wipes I was able to clean up the upper hard drive and the memory modules. Luckily the hard drives were as far apart as possible, if they had been on top of each other I don't think the upper one would have survived, I also changed the PSU out for another one that I had and also vacuumed out all the burnt bits of plastic, finally I then blew out any remaining remnants with compressed air. 

The power was reapplied and low and behold it all came back to life. No lost data due to the second dive of the raid array still working, if that had failed we did have an external backup as well but that would have resulted in some data loss due to the gap in time since the last backup.

So the morel is make sure you do your backups and make sure they are external to the device.