This site is supported by the advertisements on it, please disable your AdBlocker so we can continue to provide you with the quality content you expect.

Welcome to Our Community

Wanting to join the rest of our members? Feel free to sign up today.

Priceless backup tutorial that EVERY Windows based producer needs to read

Discussion in 'F.O.H.' started by darthjujuu, Mar 17, 2013.

  1. darthjujuu

    darthjujuu Member

    Joined:
    Nov 17, 2008
    Messages:
    1,455
    Likes Received:
    0
    Trophy Points:
    36
    Location:
    Boston
    Sup guize! :wave:

    It's been quite a while since I've perused Sneap-land, and today I make my triumphant return with an absolutely priceless tutorial in data integrity, in lieu of my own personal experience (and disasters) so that none of you should have to feel the same pain I did.

    Back story:

    I had a very thorough method in place to ensure I never lost a single take to drive failure / data loss. In addition to regular external drive backups, my studio desktop rig had two 640 GB Western Digital Blacks in a RAID1 mirror array. For those not-so-tech savy, a RAID1 array is when you have two identical hard drives that the OS recognizes as just one drive, and every write gets mirrored to both of them, so that the contents of both drives is always completely the same. The idea is that if one of them fails, Windows doesn't even skip a beat and continues to use the other drive as if nothing happened. The user is quietly notified, the bad drive is replaced, and the array rebuilds itself and you never lose a single byte.

    Sounds like a no-brainer for any studio engineer with the know-how to do some PC configuration, right? Desktop drives are so cheap these days that it's hard to find an excuse not to. I'd recommend anyone with a desktop rig (Windows or OSX) to set up a RAID1 array. There's other more complicated RAID arrays using many drives to get even more speed/data protection, but for the sake of pro audio, two desktop drives mirrored is usually plenty.

    But I digress, if you want to learn more about RAID and other RAID configurations (and you definitely should if you're interested in editing HD video) you can google or wiki it.

    The Problem:

    Notice that throughout the description of the aforementioned data protection solution, I was speaking in the past tense. That is because what I thought was a bulletproof plan, was in fact, not. Here's what happened:

    Some of my files, in this case guitar takes, became corrupted. A session that I hadn't opened in several weeks suddenly had several blank spots where there was previously guitar. I took a look at the take files, and found that a few of them were now 0 bytes. Peculiar, I thought, since I was reading these files from my mirror array. If one of them corrupted on one drive, shouldn't the good drive rebuild the missing data on the bad drive? As it turns out, one of the drives had some bad sectors, but was still partially functioning. Evidently, the corruption got mirrored to the good drive as well. I confirmed this by removing the drives and checking the contents of each of them individually. So in a RAID1 array, sometimes the corruption gets mirrored... sounds ridiculously counterproductive. Apparently, the system can only intelligently recover from and notify me of a total drive failure. I think, possibly, because I had S.M.A.R.T disabled in my BIOS settings (defaults, I have no idea why) the system was unable to foresee the failure of just a few bad sectors. The corruption was subsequently mirrored to the good drive.

    So I think to myself, no worries, surely even if both drives in my mirror array have failed me, I always have my external drive backup, and I am always sure to back up to that after every tracking session.

    Nope. :confused:

    Since the basic copying system in Windows is not very intuitive, what I used to do is just drag and drop an entire session folder over to the external drive in Windows Explorer, and tell it to over-write everything. This means that somewhere along the way, I replaced the original takes on the backup drive, with the corrupted 0 byte ones. Totally fucked. :ill:

    I made certain to always have my priceless studio data in at LEAST three places at all times, and still lost some.

    When you have a thousand dollar job with a (really good) band that drives four hours at a time to track with you, realizing that you lost some takes is an absolutely nauseating discovery. And it is so that some of you may be spared this anguish, that I have shared this tutorial with you today. Here's how I changed up my process to ensure that such a catastrophe never occurs again.

    The Solution:

    First of all, a RAID1 mirror array is still a no-brainer for any desktop user. There is simply no reason to not have it set up because of how cheap drives are these days. Just make sure you BACK UP before setting it up! And if you have priceless studio material on there, which given the context of this tutorial I'm assuming you do, BACK IT UP TWICE! (SERIOUSLY!)

    Secondly, and this is the real priceless tip, make differential backups to an external disk, using a tiny piece of freeware known as http://codesector.com/teracopy. TeraCopy is an absolutely genius, tiny, efficient piece of code. What it does is replace the shitty counter-intuitive file copy system in Windows with one that is absolutely delightful to use. And when I say external disk, I don't mean a portable hard drive, I mean a powered desktop external drive with its own power supply that isn't specifically meant to be portable, per-se. One of these should live on your desk permanently and should not be touched, because a drive that doesn't go anywhere is a drive that is less likely to fail. Use a fast interface like USB 3.0, eSata, or 1394.


    What is a differential backup and why should I give a shit?

    A differential backup is one that doesn't touch the shit that's already backed up. So say you already have your main hard drive backed up to an external. Then, you work on a session tracking all day. You don't want to completely copy over your entire main drive again, you just want to UPDATE the backup with the new stuff. But, at the same time, for a studio engineer there could be thousands of new files and an assortment of project files that have been updated, and it can of course be tricky to know specifically what needs to be backed up. With Windows alone, the only way to make sure you haven't missed anything is just to plop the whole thing over there again and have it over-write what was already there. With TeraCopy, when you drag everything over in Explorer, the first thing that happens is you are presented with a much more intuitive dialogue in regards to how you want the copy procedure to be handled.

    [​IMG]

    And all we need to do, is smack the "Older Only" button. What this does is skips the files that are already there, UNLESS you've modified them since the last backup. This means session files and things like that will be updated, but takes will not. This ALSO means, that if a file (such as one of your takes) becomes CORRUPTED on your main hard drive, the ORIGINAL doesn't get replaced with the corrupted file. This is because the file system only updates the "date modified" flag on a successful write. If a file is modified due to corruption or bad sectors, the "date modified" flag does not get updated, and thus the bad file doesn't replace the original.

    Additionally, TeraCopy offers another priceless and ingenious feature, the "Verify" button. This will run a CRC (cyclic redundancy check) on all of the files on your backup versus the originals. It checks all of them, even if you're doing a differential backup. This means if a single byte in a single file doesn't match up with your originals, you'll spot it quick, thus identifying corruption on the spot before it has a chance to spread and ruin your life. This feature helped me spot a few other files in other places that had been messed up from my mirror array. More often, hopefully, the verification will provide peace-of-mind when you're reminded that everything is definitely intact.

    So that's it! Pretty bulletproof!

    To re-cap:

    1. Get TeraCopy - http://codesector.com/teracopy - (it's freeware.)

    2. Use it to keep a backup of all of your crucial studio data on an external desktop drive. When updating your back up, use the "Older Only" button.

    3. If you aren't confined to a laptop, set up a RAID1 array. It's a fun project and good IT learning experience.

    Mazel tov! :Spin:
     
  2. mstone564

    mstone564 Member

    Joined:
    Jan 21, 2010
    Messages:
    756
    Likes Received:
    0
    Trophy Points:
    16
    This is AWESOME, dude. TeraCopy sounds like something I desperately need! :)
     
  3. Shturmovik

    Shturmovik Member

    Joined:
    Apr 23, 2009
    Messages:
    51
    Likes Received:
    0
    Trophy Points:
    6
    Saw "priceless" in the subject title and read through the whole thing waiting for the joke to happen... :dopey:
     
  4. The StabbinCabin

    Joined:
    Sep 19, 2008
    Messages:
    822
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Sacramento, CA
    Where've you been, Julian?
     
  5. tgs

    tgs Elder

    Joined:
    Dec 18, 2003
    Messages:
    582
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Stockholm, Sweden
    Good tips. Here's one from me: Allway Sync

    I've been using it for several years for backups. This also does differential backups BUT you also have the option to delete files that are not present in the original location anymore (they go to the recycle bin and all such operations can be undone).
    Deleting these files is essential for me since I always try to keep my sessions neat and tidy, so if I bounce some tracks in Cubase for example I want the old files to be removed in the backup location as well.

    I use the same program to make sure that both of my main backup drives are up to date as well: two 2TB drives, one that is the primary backup drive and one that I mirror this to about once a month.

    The software comes free as shareware but has a limitation on number of files which is far too low for DAW users, but the Pro version is quite cheap. I use this on all my computer and bring it with me for outside sessions as well.
     
  6. botus99

    botus99 Microphone Assassin

    Joined:
    May 20, 2007
    Messages:
    872
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Chicago suburbs
    This is beautiful! I just bought a couple new backup drives and was looking for just this type of information! Thanks darthjujuu!
     
  7. arvoitus

    arvoitus Member

    Joined:
    Dec 12, 2006
    Messages:
    1,230
    Likes Received:
    1
    Trophy Points:
    38
    Location:
    The Netherlands
    Good example that proves that RAID is NOT a backup.

    Your raid-controller can die, your files can become corrupt, your pc can be stolen, your house can catch fire ... all examples when raid won't save your files.
     
  8. darthjujuu

    darthjujuu Member

    Joined:
    Nov 17, 2008
    Messages:
    1,455
    Likes Received:
    0
    Trophy Points:
    36
    Location:
    Boston
    I've been out of the game for a while, but I've a few projects coming up so I thought i'd peak in on y'all ;)

    Looks sweet! Crucial for those who need to trim the fat on the backups. I, too, am an anti-hoarder when it comes to file organization.

    Definitely. A RAID1 is good to have to protect against a drive spontaneously bursting into flames, but is definitely not a replacement for a comprehensive backup solution.
     
  9. darthjujuu

    darthjujuu Member

    Joined:
    Nov 17, 2008
    Messages:
    1,455
    Likes Received:
    0
    Trophy Points:
    36
    Location:
    Boston
    Anssi adds some tips that I (as a Windows/Reaper user :lol:) have overlooked:

    "For 'complex' structured systems as audio sessions (you can spread the files to several files, folders AND drives), I suggest you backup the consolidated session, so that everything is in one places instead of doing it manually by drag and dropping, as that makes the session prone to errors when copying it from place a to place b. At least in Cubase and Pro Tools there is a session backup function, in Pro tools is "File > Save Copy In" and in Cubase it's "File > Back up Project". Just do that after a good days of work, copy to 2-3 places and archive away.

    And in OSX there is a simple backup copy system called Time Machine."
     
  10. Direngreyyy

    Direngreyyy New Metal Member

    Joined:
    Mar 5, 2013
    Messages:
    17
    Likes Received:
    0
    Trophy Points:
    1
    And it's free too :)
     
  11. KillFrenzy

    KillFrenzy Member

    Joined:
    Nov 21, 2007
    Messages:
    403
    Likes Received:
    10
    Trophy Points:
    18
    I've been using SyncBack Free for about 4 years now. Version 6 looks a bit strange, but the older version is PERFECT for my needs.
     
  12. Benedict16th

    Benedict16th Benito Slipperman

    Joined:
    Dec 23, 2012
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    You can also use Gobbler, backup cloud storage build for audio projects. It monitors and backs up data changes and you can restore previous file versions.
     
  13. slo77y

    slo77y Member

    Joined:
    Aug 3, 2010
    Messages:
    176
    Likes Received:
    0
    Trophy Points:
    16
    i use rsync through cygwin. its the most powerful copy tool i can think of. i copy directly to my raid5 backup server :)
     

Share This Page