Skip to content

Public Service Announcement: Make sure your backups are up to date

I am sure this has happened to more people than would care to admit it: Your clients’ (or your employer’s) backups are running smooth and you correct any hiccup right away but you are lazy about your own backups. So I start my (somewhat shameful) and hopefully eye opening post.

A few days ago I saw a Windows Delay Write Failed notification in my taskbar. I think most of us know it, this notification usually means a write operation failed, and if your [external] hard drive is plugged in, on, and should be working fine, you start to suspect that maybe the drive is failing or was somehow unplugged before data in the write queue was flushed. But I digress, the notification came up, I then proceeded to access the logical volume in question (which consists of a LaCie Big Disk spanning two 500GB disks on a RAID 0) and was able to access some of the files just fine. I then attempted to run a chkdsk on the drive but I was met with an error stating that the type of the file system was “RAW” and thus chkdsk was unable to scan it. This my friends, is when I started to panic.

Normally, I wouldn’t be worried because I knew that I had a backup from the night before, however, recently the hard drive (also 1TB) that I use to backup my data started malfunctioning culminating in an RMA. Once I received the new drive I procrastinated mainly because I was going to purchase a Sans Digital RAID enclosure which I planned on using as a new volume for all my data (consisting of a pair of 2TB disks on a RAID 1) soon. However, I was worried because I now had no viable backup of my data and for all I knew one of the two drives on the RAID 0 array had just gone bad.

I quickly prepared the replacement 1TB drive on one of my external SATA enclosures and prepared it to copy all my data using Robocopy over my Gigabit network (this would normally take ~12 hours for a full backup) but knowing that the file system was not being read properly I decided to try my luck in power cycling the LaCie Big Disk, boy was that a mistake! The drives wouldn’t spin up again.

I proceeded to “freak out” for about five minutes, actually having to take a few minutes away from my desk in order to get past the fact that I might have just lost 600GB worth of data. Once I was able to sit back down I proceeded to go through the basic troubleshooting steps:

  1. Take the enclosure apart (voiding the warranty, but at this point, I didn’t care about the warranty as much as my data)
  2. Re-seat the SATA power and data cables
  3. Disconnect each drive individually and see if the other would spin up

Everything I tried failed, things were looking grim.

Having another SATA enclosure on hand, I removed one of the SATA drives from the LaCie enclosure and installed it in my other enclosure, this time the drive spun up! I quickly confirmed the second drive spun up as well using the same method. By now at least I was sure that the drives were spinning so I started to feel some of the weight off my shoulders. My next step was to find out how to recover data from a broken LaCie RAID. Searching Google found a few noteworthy results and at least I was on my way to a real solution. While searching I heard about a few tools, R-Studio and iRecover being two of them, I would try them as soon as I had corrected my backup situation.

I also found the likely culprit for the problem: one of the blog posts I found online while searching stated that their problem was sourcing from the power adapter that the LaCie drive uses. The writer wrote that there was a ‘hissing’ noise emanating from the power adapter noticeable when your ear was placed on the adapter – I was able to reproduce this noise and confirmed that none of my other power bricks (albeit, different power bricks) were producing this sound when my ear was placed on them. The fix could have easily been to purchase a replacement power adapter, but this is the second Big Disk failure I’ve experienced so I will not be giving any more of my business to LaCie.

I spent the next 20 hours or so using WinImage 8.1 to create VHD images of each drive, the process would have been faster but I didn’t have enough space on the computer where I was making the image so I had to create the image and save to a network location (which I can only assume slowed the process down a little, though when I think about it maybe not so much as the USB bus since my Gigabit network should outperform the USB bus being used to read from the drive). Anyhow, after both VHD images were created I now felt that I had a viable backup of my drives and data. I put the two 500GB drives in a safe place and got to work on finding some tools that could “de-stripe” my array. Thanks to Windows 7’s built-in ability to Attach VHD files as local disks, I was able to perform all my tests on “Read Only” copies of the disks as if they were directly attached to the system (which was awesome).

The first tool I tried was Runtime Software’s RAID Reconstructor.  This tool was unable to detect the RAID parameters that LaCie used so in the end I couldn’t use it (and it wasn’t until later that I figured out why). The default detection settings that RAID Reconstructor used were out of the range of the RAID array start sector and the array block size was not one of the default settings the software provided you with (this LaCie drive used a 256KB block size, where the largest block size that RAID Reconstructor specified by default was 128KB). The parameters to search for the RAID array are fully customizable but I was unfamiliar with RAID parameters in general and the software’s RAID detection feature was unable to find the array parameters with the settings that I provided in addition to its own recommended settings so I uninstalled the trial and moved on to the next piece of software.

The next software I installed was iRecover by DYI Data Recovery. This software was easily able to find my array and reconstruct it, however the scan time was about 30-40 minutes. I was also unable to save the scan results so I would have to reconstruct the array every time (to be fair, it seems the registered version allows scan results to be saved). Once the array was reconstructed I was able to save a large file and confirm that it was not corrupted, great news! I was now ready to purchase iRecover, which costs $90, but first I wanted to try the last option that my search yielded: R-Studio.

R-Studio offers a powerful tool set for the technician with the R-Studio Local and R-Studio Network products. You can find out more about their products on their website. I downloaded R-Studio Local and got to work, I followed the RAID Recovery Presentation which provided me with enough information about the RAID parameters that I felt like I could do more than I could previously with RAID Reconstructor and within a few minutes was looking at my data, no scanning required! I will admit it took me a few tries to get the block size correct. I had a hint about the drive order (one drive would show as unallocated space while another came up as two partitions on the R-Studio GUI) so I used the drive order that I thought was correct at first (the drive with the partitions first, then the drive with unallocated space) and varied block sizes until I found the sweet spot: 256KB.

Armed with my tool of choice: R-Studio, I proceeded to recover the data from the two 500GB drives attached to my system via a pair of USB SATA enclosures (to avoid recovering from the VHD image across the network, though honestly I think my USB bus is taking a harder beating than my network would have taken) to a third 1TB disk attached via a third USB SATA enclosure. A few hours later my data is still being recovered but this goes to show that being lazy and procrastinating can bring you a lot more work in the long run.

I still haven’t purchased the Sans Digital enclosure I want, and while I will probably do that sometime next month, as soon as the data is recovered I will be re-scheduling my old Robocopy job to run every night and mirror one drive to the other over the network.

Never a dull moment.

One Comment

  1. Robert wrote:

    I think the bad issue was the I/O unit. I’m doing some test with a 2TB (perfectly working, just testing). R-Studio says I/O unit is 512 bytes only. Something I could not set into raidrecon ’cause it just starts from 1Kb.

    Wednesday, June 2, 2010 at 11:36 am | Permalink

Post a Comment

Your email is never published nor shared. Required fields are marked *
*
*