Back That Stuff Up: A Cautionary Tale


For the past few years I’ve hosted my sites on a VPS (Virtual Private Server). I trust the hosting company and like their service. In four years I haven’t experienced any downtime and the tickets I’ve opened were answered in a timely manner. This setup left me very comfortable with my hosting. Cue foreboding music.

Last Saturday my web host sent me an email you generally don’t want to see. It included phrases like “…failed RAID array…”, “…bare metal restore…”, and “…disaster recovery…” I say they sent me the email since I didn’t receive the email until the next day. All of my email is funneled through my server which was at that time was tango uniform. I had Pingdom setup to alert me if the site was down, but these alerts were also being sent through my email and I had used all of my free account’s SMS notices.

Friday night I was enjoying the start to my weekend when someone who was trying to read one of my blog posts alerted me to the error (Thanks Glen!).

I checked and my server was totally offline. I opened a ticket with my host who let me know of the outage. My server would be down for a total of 12 hours after which it was restored with week old data. My wife and I had been blogging more than usual that week so we had a week of data that would need to be recovered some way. Damn.

I’ve since rebuilt the missing posts and I lost some drafts, comments and comment replies. It took me an hour to do the rebuild, but it could have been much worse.

Here are five things you should think about when you plan backing up your site.

1. Outages happen

In 2009 I hosted at a company who had an explosion in a service closet that melted servers and took affected sites down for 4 days. Thankfully this outage wasn’t that severe.

All hosts will have downtime. The severity of the outage will differ between hosts.
Later Saturday night there was a managed WordPress host that had an outage. Earlier in the week a large shared hosting provider went down (again). Since we can’t avoid outages the best thing to do is make sure there is a plan for surviving them.

2. Have timely backups in place

Key word: Timely. I know there are several plugins that will do site backups for WordPress, and some hosts offer backups as part of their service. I use InfiniteWP to manage my WordPress sites and back them up to an Amazon S3 bucket. I had a backup scheduled weekly…on Sunday nights. So when the outage happened early Saturday morning I had a backup from a week earlier. Edge case hammer, meet my forehead!

Also, I had started hosting a blog for a friend over the holidays that I had not yet added to the backup schedule. He and his staff had been writing posts and adding drafts like mad. I had no backups for his site. Thankfully most of his data was accessible via Google Cache, but there were some very long drafts that were not recovered. I felt terrible.

To alleviate this edge case I switched my backups to daily so that the most data loss I’d incur would be 24 hours instead of 7 days.

3. Think through being notified

As I mentioned earlier my site monitoring strategy consisted of a free Pingdom alert with no SMS credits. There are plenty of monitoring options out there, but the key is to think through the process: “Server goes down. I will get notified by this alternate route that doesn’t depend on server being accessible.”

4. Backup your editorial process

All of my blog posts were recovered because I had copies of the posts in my email from a Mailchimp Email to RSS subscription. But, that doesn’t mean I didn’t lose data.

My editorial process is this:

  • I use the WordPress Editorial Calendar plugin to create post draft on the day I want to publish a post
  • I’ll add a descriptive title and some notes in the content to remind me what the post will be about
  • I’ll write the post in Google Docs then copy to WordPress when it’s time to format, add a picture and schedule the post

I had spent a few hours during the week planning out the entire month of posts for two of my blogs. With the outage all of those post ideas and scheduling went out the window.

I’ll probably go back to having my editorial calendar and drafts in Google Docs or Trello. The lesson here is that you should frequently backup anything that you spend your time on and is valuable to you.

5. Decouple email if you want it to stay online

I had already been thinking about decoupling my email from my server. I currently have email running through my server with all addresses forwarded to a Gmail account. This created a single point of failure and when the server went down so did email. No email turns me into a sniveling diaper baby.

To make sure this doesn’t happen again I’ll be moving to something like Google Apps, Zoho Email, or FastMail.

Other than the typical “Back that stuff up” do you have any lessons learned from losing data?