Last week Jon wrote about the horrifying experience everyone eventually has to deal with – data loss. His wife’s hard drive failed – and without a current backup to rely upon – he went through two data recovery companies and two thousand dollars to retrieve 80 gigs of home movies, pictures, and a career’s worth of graphic design files.
Jon’s lucky. He got his data back. My sister wasn’t so fortunate. I got a frantic phone call from her last night (which is why I’m writing this post). She kept a shortcut to her pictures folder on her desktop and somehow managed to delete the real folder when she dragged it to the trash. What did she lose? Over three thousand pictures she had taken from her senior year in highschool and first year at college. Gone.
I’m a paranoid person when it comes to backing up my files, but it took a hard drive crash of my own to make me realize how important backing up can be. Since then, I’ve gotten very good at making sure my data is secure. The setup I’m about to describe works for me. It’s based on how I prioritize my data and on the budget I’m willing to spend to keep everything safe. It’s not perfect for everyone, so take what I say with a grain of salt – an example of where to start and what’s possible.
All of my data falls into six categories.
- My Home folder (pictures, documents, application preferences, etc)
- My iTunes library
- My Movies folder
- Websites I’ve built (both personal and professional)
- Applications
I segregate my data this way based on how frequently I back up each and the method I use to do it.
Keeping my email safe is by far the easiest to do. I have two accounts that I check. My GMail (personal) and my Sitening work email.
The philosophy behind Google’s email service is that they give you so much space, there’s never a need to throw anything away. I try my best to take advantage of that. I switched to GMail in May of 2004, two years ago, and every email I’ve received or sent is at my fingertips on any computer with a quick Google search.
But what if Google crashed? I can’t leave my mail in one place. Luckily, Google allows POP access to GMail which means I can save it onto my local machine very easily. Using these simple setup instructions, Apple Mail keeps a backup copy of all my mail.
As for my Sitening email, the solution is even simpler. We host our website at TextDrive which means we get IMAP email access. IMAP lets me keep an always-in-sync copy of my mailbox on my machine and on their server. If either should crash, I can restore one from the other. Simple.
My Home Folder
The beauty of OS X’s Unix-based design is that each user gets their own Home folder where all of their data is (should be) stored. In theory, you could make a complete backup of your Home folder each night and be set. Some people do that. However, for me, that just isn’t practical. I’ve got too much data.
Here’s how my Home folder breaks down:
- Documents folder: (600mb) Contains my financial files, lots of ebooks, my resume, all the papers I wrote in college, back-up files from previous jobs, etc.
- Library: (600mb) This is where OSX programs store your preferences (specifically in ~/Library/Application Support and ~/Library/Preferences).
- Movies: (12GB) Home videos and movies and TV shows I’ve downloaded.
- Pictures: (4GB) All of my digital photos and lots of old family pictures I scanned.
- Music: (25GB) A large portion of my music library.
Add that up and you get 42GB of data!
So what do I do?
In the corner of my home office I’ve setup my old Dell desktop to be a simple file server. I replaced its aging hard drive with a new 20GB that I’ve installed Ubuntu Linux on. Separated from the OS is a second 120GB hard drive which I use for storing my backups. The drive is shared to all the computers in my house (five at this point). Total cost? $150 for the two hard drives, $0 for the operating system, and I’m not counting the cost of the computer since everyone seems to have an old machine laying around now-a-days.
Next, I add Shirt Pocket Software’s SuperDuper! to the mix. It’s a fantastic little OSX app that does intelligent backups. For $27, it will scan any folder(s) you tell it to and do either a full backup or just backup the files that have changed since last time. Best of all, it lets me ignore certain files that I don’t want to include.
Here’s a screenshot of the main SuperDuper! window
and of the options I’ve set.
As you can see, I’ve told it to back up my entire Home folder except for my Movies and Music folders (we’ll get to them later). Each day when I come home, I set my laptop on my desk, launch SuperDuper and go start dinner. Five minutes later (thanks to it’s smart backup system) my entire Home folder is backed up onto my file server.
Movies Folder
With 12GB of movies on my laptop, how do I decide what to backup? Easy. I burn my home videos to DVD twice. (Store one copy at home, the other at work, or with a family member, or even a safety deposit box at the bank). The rest of the movies? The TV shows? Who cares. There’s no reason to keep around month old episodes of Lost or 24. The same goes for any movies I’ve downloaded. If they get lost (no pun intended) it’s no big deal. (Obvisouly some people make take issue with this point. But like I said, this is my backup solution. YMMV.)
iTunes Library
Backing up my iTunes library took a large initial investment of time. I made sure that all of my files were tagged with the correct album, artist, and song title information and then told iTunes to burn the entire library to data DVDs…twice. My 25GB collection spanned 6 DVDs. (I could have used larger, dual-layer discs but those are too expensive.) Once the backup was finished, I stored one set of discs at home, the other went to my parent’s house.
That solves the inital backup problem. The trick is to continue backing up new music as you aquire it so you don’t have to do a huge multi-disc backup each time.
To do this, I created a smart playlist in iTunes called “New Music” with the following rule.
This creates and up-to-date list of all the music I’ve added to iTunes since I last backed up. When the playlist reaches 4.7GB (the size of a DVD) I burn a copy and reset the smart playlist’s date to today. It’s an elegant solution that works really, really well.
Websites
Building websites is how I make my living. It’s vital that everything I produce is backed up and in some form of version control.
Every website I build is stored in SVN (on my local computer, and also on a remote server we use at Sitening) so I can roll-back to any previous version of any file at any time.
For an extra layer of protection I schedule SuperDuper! to make a backup of my websites each time I do the Home folder backup mentioned above.
Applications
Finally, what to do about all of the applications I use.
I thought about this problem for a while and realized that backing all of them up is not something I want to do. That’s way too much data – especially considering how often new versions are released.
I started looking at exactly what was irreplacable. Very little it turns out. Most of the programs I use are small and can be re-downloaded at any time from company’s website. For example, TextMate, SuperDuper!, Transmit, NetNewsWire, Firefox, etc. The important thing is that I have my registration number handy for each one. All those codes are, of course, stored in my email which is already backed up. Sweet.
As for the larger applications like Photoshop and Office, well, I just have to bite the bullet and makes copies of their install discs and store them someplace safe.
Still, backing up my applications isn’t quite finished. In the horrible event that I have to recreate my system from scratch, there’s no way I’ll remember all of the little programs I use on a less than daily basis. I need a way to keep track of what’s installed on my machine.
To keep track of this I wrote a small shell script that runs once a week.
ls -l /Applications /Applications/Utilities > myapps.txt
mail -s "List of Programs on Tyler's Laptop" you@somedomain.com < myapps.txt
This prints out a list of all the programs on my machine and emails it to myself. Now, if my system were to crash, I have a complete list of what to download along with the registration codes in my email.
The End
Like I said at the beginning, this is my backup solution. It's far from ideal (I'm always tweaking things), but it works for me. From talking with friends and other computer people, I've realized that everyone's needs are different. The important thing is to make sure you find a solution that works for you, and stick with it.
Filed under Data Recovery, How To, Linux, Productivity
Comments
I have tried dozen of back up softs
Super Dupper is only the most reliable one
And Dave does real good hotline for OSX pb too
Best
Frederic-Paris-France
For any backup needs, give Dobry Backuper a try:
http://dobrysoft.com/products/backuper/
Dobry Backuper is a reliable and easy-to-use backup application, which offers most of the features of expensive backup software for a fraction of the price ($30).
Dobry Backuper can archive any files, including hidden Unix files to any mounted device (i.e. a network volume, iDisk, an external drive or even iPod), or to a number of CDs and DVDs. The program offers several backup types – full, incremental or versioned, either with or without compression. The archives use a tar-compatible format, meaning data can be restored even without Dobry Backuper if needed. Operations can be scheduled so that backups are made when needed not requiring your attention.
It’s enough to create and configure your backup only once, and the program will remember about it, automatically building and saving it regularly.
Excellent article and discussion, Tyler and commenters. Thanks!
You might also consider the issues discussed in these two articles:
http://blog.plasticsfuture.org/2006/03/05/the-state-of-backup-and-cloning-tools-under-mac-os-x/
http://blog.plasticsfuture.org/2006/04/23/mac-backup-software-harmful/
(BTW, I have no connection to any hardware manufacturers, software developers/authors/publishers, or blog writers. I’m merely an ordinary Mac user who needs to maintain backups of the highest integrity.)
Personally I find iMsafe (http://homepage.mac.com/sweetcocoa/imsafe) really useful – it does incremental backups, it is easy to tell it exactly what to back up and when, it autoconnects to network volumes and it has a pretty nice, clear interface.
I second the use of superduper – what I do is partition a network attached drive in two, then do alternate backups weekly onto each drive. If I delete a file before the backup, and backup, I then have a previous version of the backup on the other partition to hand if need be.
Superduper allows you to make a drive bootable after backup, so that one can boot into it if need be!
“what I do is partition a network attached drive in two, then do alternate backups weekly onto each drive. If I delete a file before the backup, and backup,”
Aha! Now that’s a clever way to make sure your files don’t get corrupted during backup without having to maintain a ton of extra copies. Nice.
tyler, do you have any experience with the laCie mini companion hard drive? I just bought it, and have installed it, but I have no idea how to backup my Powerbook hard drive on to the external one. any ideas?
“Tyler, do you have any experience with the laCie mini companion hard drive? I just bought it, and have installed it, but I have no idea how to backup my Powerbook hard drive on to the external one. any ideas?”
As long as what you bought is just a typical external harddrive, just about any Mac backup program will let you backup your files to it (or create an exact copy of your existing hard drive). Most of my experience is with SuperDuper! and ChronoSync. Both of them should work fine for you. Visit their websites for FAQ’s on how to set them up
I use Apple’s free Backup app with my .mac account. Its so easy to create different backup stratagies and for $99 bucks its well work it.
all,
in addition to my regular superduper! cloning routine, i’d like to be able to create an incremental backup system using gmail.
i created a little automator action that “zips” any recently changed files that i have labeled “red.” typically my files are pretty small text files .tex/.cpp, etc. so the zips are plenty small for email. occasionally there might be a word document or something.
i name the zip files by date so they can pile up in a folder for a while, but next i would like to email the zip to my gmail account.
using mail.app still requires user interaction (yes?) to send the email.
it seems sendmail/mail doesn’t like zip files.
is there a simple command line utilty or a way to get mail.app to run in the background without intervention from me?
it would be great if i could this going a few times a day and not have to think about these incremental back-ups.
thanks much for any advice, tips, pointers.
Following the impression I got of SuperDuper with this article. I purchased the software after having some unpleasant experiences with my PowerBook. First the Hard disk crashed and then after about 4 weeks the power book got stolen. In the first instance, I spent $900 to get at least my family photographs recovered. I backed up my photographs and whatever was recovered of the data on DVD, the power book got stolen. For the second, I am keeping my fingers cross about the likely hood of getting my PowerBook back in one piece. So, until that happens life still has to go on….
Soon after I got SuperDuper I tried backing up my files as instructed… I wanted to burn my back up straight on to a DVD but that was not possible with this software. Would it be some way to carry through with what I was intending to do….
Thanks…
“I wanted to burn my back up straight on to a DVD”
When SuperDuper backs up your files, it creates a disk image (.dmg) file which you can then copy to a DVD by inserting a blank DVD and dragging the file onto it.
It appears that SuperDuper does not make a back up disk image on the same drive that is being backed up. It does not allow it even in the case where only a few folders are being copied.
This is a problem when I don’t have a set up such as the one explained in this article with a server drive or external hard disk. I am hoping that the next version of SuperDuper takes that into consideration. Or I will need to look for a backuping software which also burns a DVD at the end of the process. Until then, I will have to manually copy my files for back up as I have been doing.
FiL,
you are wrong. you can write to sparse image anywhere you want.
if you have issues, email shirt-pocket-watch support. that is what you are paying for. their support is the best i have ever seen.
Tyler, how do you have your ubuntu server set up? Is it Ubuntu server? IT would be great if you could followup and post about your ubuntu server.
Renegade,
I’m running Ubuntu 5.10 on a 1.2ghz Dell desktop with 256mb of RAM. The OS is installed on a 40gb drive and I’m using a second 120gb drive for storage. The box itself is sitting in a corner of my office out of the way – my only access to it is via SSH, FTP, etc. I picked the corner it’s in so I could drill a hole through the wall to run ethernet cable directly from my router in the hallway closet.
My setup doesn’t have too much installed. It’s just the base system (no KDE or Gnome) with SSH, FTP, Samba, and PHP/mySQL (for a few cron jobs I run). I’ve setup the entire secondy drive as a Samba share.
Any other specific questions?
Very interesting.
I expect to use this information for myself and provide the link to my MAC user friends. The trouble with using Apple Macintosh data and data storage systems in Apple’s habit of dropping good programmes without warning or recourse- entire volumes of hyper card data are inaccessible to newer Macs. Ditto for Appleworks and …
There was some information in the article about the data recovery. Several times i have seen people loosing their data or throwing their hard disk just because they had no idea that the data in thier crashed hard disk can be recovered either by the usage of software or using a data recovery service companies like the Disk Doctors Labs Inc or the Drive savers where most of the data is recovered most of the times though the recovery success rates is not 100% percent but data is recovered most of the times in most of the cases.
Thank You!
BTW, here’s another solution for someone uneducated in Scripts like myself. To create a PDF list of apps, I used PrintWindow from SearchWare Solutions instead.
Again, thanks!
Very useful – am havong trouble running the script for apps – what is the easiest way to do this? Can you use Automator?
Thanks!
Jon
My 117 (120?) GB LaCie came with SilverKeeper and that is what I have been using to backup 20 GB iPod and to restore after the iPod crashed two weeks ago. I read about and downloaded SuperDuper and the first thing it said was it would format (erase) my LaCie drive, which was not what I wanted. My practice is to maintain multiple backup sets in a sequence, since traditionally the backup has been only 2 GB and there is almost 100 GB free space still. The LaCie is firewire and is attached to iMac, which has iPod attached to it. New iPod is 100 GB hard drive.
Hi,
for Applications, you can use ~/Applications for non Apple ones and /Applications for Apple ones, so they are not mixed. Valid if your are the only user on your system.
For people using rsync, do not forget to use option -E to preserve resource fork.
N
This is a great page. And the contents are really that worth reading. I will add this to my own library
Thanks for the advice this is a wonderful resource i have over the course of the years started and stopped too many times to remember which as left my terabyte + of hard drive capacity scattered and i am never sure of what i have backed up or not is there a simple way to sift through all the hundreds of gigs of digital image files making cd or dvd of my files is much to time consuming not to mention, a typical days photo-shoot will leave me with 4-20 gigs of data i should not be storing all of it but after 20 years of shooting slides it is hard to throw it away
Peter,
The only solution that comes immediately to mind is using Apple’s Spotlight service. Spotlight is the “instant search” feature built into OS X. If you have OS X installed on all of your machines (and they’re all connected to your local network), you can use Spotlight to instantly search across all of them. It’s not an ideal solution, but it’s the only one I can think of at the moment.
Great read. Lots of great tips. Considering Silver keeper as my back up software.
This is the best backup structure I have found. Congratulations! I’m using the same functions as you.
I just want to make a little tweak. Instead of mailing the list of applications to myself I want SuperDuper! to run a shell script and save the .txt file inside the HDD and THEN start the Backup, so the file will be backed up.
BUT! I don’t know how to create/write an applescript. Can anybody tell my how?
The command is the same as Tyler Hall’s:
ls -l /Applications /Applications/Utilities > myapps.txt
Matias – You use Script Editor. Open a new window and type
do shell script “ls -l /Applications /Applications/Utilities > myapps.txt”
FWIW, this will create the myapps.txt file at the root of your drive. You might prefer:
do shell script “ls -l /Applications /Applications/Utilities > ~/myapps.txt”
Then, save the script where SuperDuper! wants it — I haven’t used the program, myself. That’s all there is to it.
Since you’re a TextDrive customer, you might have StrongSpace. Since my Subversion repositories are the lifeblood of my business, I back them up off site to my TextDrive account nightly. Highly recommended and easy.
ive noticed the best trick to a dead harddrive and no available backup whatsoever (because I was lazy) is to order the exact same hard drive board and replace it. so far its just cost me 98 dollars and my harddrive works again (at least to copy all the data off the unreliable disk and onto a new harddrive). this is the best CHEAP plan b solution (to drive savers that is) if your harddrive dies and have no backup whatsoever.
Of course, just stop what you’re doing and send it directly to drive savers if you hear your harddrive making seriously messed up sounds.
why not mention bookmarks?
good article though
now can you back up your keychain?
i am planning to format and re -install my powerbook
not sure i need to but i haven’t done it at all since i bought the machine new in 2005…
what is the easiest way to do this.. ?
Im a new macbook user (osx 10.5) and was exploring how to backup efficiently when I came across your article. I like how you group your backups by category and it made me realize that I dont need to backup my programs (habit!). Like you say, new versions coming out all the time!
Burning movies to dvd twice seems like too much work. HD space is cheap now and TB hard drives are more common.
I think the most important thing is to create a full backup of your hd once you have everything on it the way you like…then incrementals for the stuff that may change (especially music and photos). Here a simple sync to an external drive would do.
anyhow, i used to be a long time windows nut, but I have seen the light!