Do CIOs Get Backups?

Posted on September 2, 2011 | Leave a comment

Sitting in a press room listening to a panel at an industry event reaffirmed my skeptical view of management. I dunno, I guess I don’t really expect or want a CIO to know the nitty gritty about how their stuff is backed up, but I didn’t appreciate the responses that any of them made to the question ‘how are you backing up your virtual infrastructure?’.

Each of the CIOs from the keynote session gave weak answers. They would have been better off saying that they have not had a loss of data due to any failed restores rather than just making up stuff.

The first response was given with an accompanying look of confusion as to why the question would even be asked, ‘why, we merely replicate all our VMs, so there is no problem with restoring’. Ouch, where to start with that one. So, you can successfully restore your corrupt or deleted files that successfully replicated to your other site? Maybe there is more involved, maybe he meant to say that replication, on its own, is not a backup, so we use a CDP (continuous data protection) solution to capture all the changes and replicate those offsite. Or, we perform some form of snapshot and replicate that offsite. Without some form of rollback capability, replication may fail to restore you to the state you require. Replication, on its own, is not a backup solution, since any corruption or accidental deletion would simply get replicated.  As well, snapshots, on their own, are not a backup solution, since a loss of the appliance will mean a loss of the snapshots, so replicating those snapshots to another appliance is necessary.

The other response that got my hackles up was, ‘the beautiful thing about VMs is that they are ultimately just a single file, so it is really easy to just back that file up’. So, all of you who are experiencing challenges backing up your virtual environments must be missing out on that fact. They’re just files! They back up super easy! Right? Who says CIOs don’t have a sense of humor? If it were that easy, there would be no third party solutions needed and no agents or APIs for backup products, we would just perform regular file level backups with whatever backup product we have on hand.

I don’t recall the other responses, but they were equally useless. Again, silly to expect a CIO to know how things are done, but I thought that was why they were chosen for the panel, so they could explain how they are doing things with their virtual environments. You could look at it this way, they don’t know because there hasn’t been a problem restoring, so they haven’t needed to dwell on the how. Sure, let’s go with that.

→ Leave a comment

Posted in Backup Stuff, Reed's Rant

Do You Buy Stuff Configured With The PSMO Option?

Posted on August 24, 2011 | 1 Comment

I was introduced to the concept of PSMO at a client a number of years ago.  He told me he hired me as a consultant to help him bypass the built-in PSMO feature (though he used a different letter than “S”).  I naturally asked him, “what in the world is PSMO?”  He told me that all software comes with the ‘Please Screw Me Over’ (PSMO) option enabled by default.  He expressed the opinion that you need to decipher the correct combination of configuration and/or licensing to get any product to actually work in a reasonably intelligent fashion.  And you people think I’m cynical!

I do have to agree that it often seems difficult to get software to work as you would expect, or figure out which of the numerous options you have to license to make things work like the sales guy told you it would.  I’ve done ROI’s that looked great until you added in the software license to enable one critical feature, after which you just couldn’t get people to even consider the solution due to the now exorbitant price.  While I understand the greed and avarice behind licensing prices (wasn’t it the Beatles who sang “All You Need is Cash”?), it does seem like they are trying to emulate Steve Martin’s pseudo philosophy on his Wild and Crazy Guy album.  To paraphrase Steve, these companies might say “this is what I’m shooting for – one sale, goodbye”.  Don’t get me started ranting about programmers creating programs that have complex configurations or options buried or obfuscated in some bizarre way.

At Tech Field Day 7, I heard some interesting comments from the sponsors that may indicate a growing tendency to move away from this long running problem of PSMO.  I’ll be blogging separately on the products that the sponsors talked about.  In this post, I’ll focus on the licensing side of PSMO, since we didn’t really get into configuration details.

Symantec

Symantec has bundled a number of their options with their Backup Exec appliance.  This should help simplify things.  Clients can be pushed from the appliance, which is a full media server.  I think it’s lame that Symantec didn’t include the Central Admin Server Option (CASO).  The assumption being that this appliance is for small environments, who may not need CASO, I suppose.  However, to keep from moving too far away from the PSMO philosophy, there is no ability to send to tape with this appliance and no RAID-6; so unless you’re ok with your appliance having a double-disk failure and losing *all* of the backups sent to that device, then you will need to have CASO to duplicate these backups to another media server to have a second copy on disk or tape (would have to be a regular media server not an appliance for the tape copy).  Fear not, I plan to cover these deficiencies more in another blog.

Symantec also covered their NetBackup appliance line.  The software and hardware licenses for these devices are separate on the NetBackup side, as opposed to the Backup Exec side having them tied together.  The nice part about the licensing here is that there is no additional software licensing for replication, simply add the hardware for another appliance.  Symantec moved to a capacity-based license several years ago, though not everyone knew about that when it came out.  In many cases, it is cheaper, and way easier, to have a capacity-based license.  You pay an amount based on how much data you are protecting in the environment and no longer care how many clients or options you need, they are all included.

Dell

Dell’s move away from PSMO is to have a perpetual license.  I wasn’t planning on blogging on this subject, so I didn’t really take detailed notes on licensing stuff.  I may have missed how far they are planning on taking this.  I have written down that a licensed purchased for a Compellent array stays with you, even when you do a fork-lift upgrade to new hardware.  It is possible they will do this with all their hardware.  So, buy it once and it follows you forever, like rats following the Pied Piper.  That’s probably not a good image for licensing, so forget I said that.

Veeam

Veeam has a good approach on the licensing side of things.  They now have a single license for multi-hypervisor environments.  So, if I decide to start moving to a hypervisor other than the pervasive VMware (not that anyone would ever accuse VMware of moving to a more PSMO-like licensing scheme anytime recently), I can use my existing licenses to say move one of my servers from VMware to Hyper-V.

My Shining Ray of Darkness

I certainly could twist this to say that this is just an example of the fact that licensing prices are so over-inflated that they can easily afford to throw more crap in and it won’t cost them anything.  But, I won’t say that.  I did get the impression that the companies above are making a genuine effort to make your life easier and trying to provide PSMO-free products.  Making your life easier, after all, would have the benefit of you feeling more inclined to turn to them and buy more stuff.  I also think it’s ok to ask vendors for products that have the PSMO option disabled at time of purchase.

→ 1 Comment

Posted in Reed's Rant

Tagged , ,

FCoTR Takes on TSA

Posted on August 13, 2011 | 4 Comments

I have proof there is a government conspiracy against Fibre Channel over Token Ring (FCoTR).  I arrived at the Austin airport Saturday morning.  I was minding my own business, as I always do, (unless, of course, I’m minding someone else’s) when the metal detector at security went off.  I never set that thing off, I’ve done this way too many times to forget some stupid thing that would set that off.  I pat myself down several times and still come up empty.  Since I’m still setting off the alarm on the detector, I get sent to the special cow pen for the full pat down procedure.

Here’s what they found in the bottom of one of the 20 pockets in my cargo shorts:

FCoTR can't get past TSA

Just an example of the conspiracy against FCoTR.  Don’t believe any claims that this is just a protocol issue, such as the token being passed down the wrong path or excessive delays in granting permission for transmission, those types of comments are just diversions funded by the Ethernet Lobbyists (aka Communists).

I’d like to point out that my reason for being harassed at the Austin airport was due to my attending Tech Field Day 7.  It was my first time attending and I had a blast.  I learned some good stuff and met a bunch of cool/geeky people, if you’ll pardon my putting those two terms together.  Stephen Foskett and Matt Simmons put on an excellent event.

P.S. My > 3 oz bottle of Maple Syrup, courtesy of Mr. Foskett, made it through TSA in my carry-on just fine, much easier than the little FCoTR pin.

→ 4 Comments

Posted in Reed's Rant

Tagged , ,

Easy Hard Drive Upgrade with Mac

Posted on August 7, 2011 | Leave a comment

I’ve been limping along with my little 128 GB SSD drive on my MacBook Pro.  (That statement still seems odd to me considering that my first hard drive upgrade, ages ago, was to a 30 MB hard drive that my sister told me I’d never be able to fill.)  My kind boss felt sorry for me and sprang for a new, roomy drive.  I received my new 500 GB SSD drive and went straight to swapping hard drives, which was incredibly easy.  The steps I took were:  1. made sure my backups were current, 2. swapped hard drives, 3. restored my OS using Time Machine, 4. reinstalled my Windows Boot Camp using Winclone and then 5. checked to make sure everything was working.

Step 1: Ensure Backups are Current

This could be as simple on a Mac as clicking on the Time Machine symbol on your menu bar:

I have an external drive plugged in, so backups are updated every hour.  I verified that mine was only a few minutes old, then checked on my external drive for a current Winclone backup of Windows running on Boot Camp and then verified that I had an up-to-date backup with CrashPlan, ’cause I’m just that paranoid.

Step 2: Swap Hard Drives

I use ifixit.com for things like this, they have nice step-by-step instructions, complete with pictures.  For my laptop, I used this guide, http://www.fixit.com.

The oddest thing I ran across was the bizarre screws (Tri-wing Y1 screws) used to attach the battery, I’ve never seen them anywhere before.  Even my cool little screwdriver with combo phillips #00 and T6 torx bits doesn’t have that.

Moving slowly and being careful not to touch the motherboard (since I couldn’t remove the battery), it still only took me about 10 minutes to swap the drives.

Step 3: Restore OS with Time Machine

This part is super easy, only made difficult by the fact that we have 4 different MacBook Pros in our office and, apparently, they each have their own distinct install disc.  Yes, this could have been made a non-issue by taking 10 seconds to use a sharpie and write some comments on each disc.  Since we didn’t do that, I grabbed what I thought was the most logical disk to use.  Silly me, thinking I could just use a Snow Leopard disc.

I did learn a new way to eject discs; when Command-E and the eject button don’t work, you can push the track-pad button while rebooting.  Turns out that rather than saying “this is the wrong disc, idiot”, Mac simply decides to keep the disc and not allow normal eject methods to work.  Kind of like what a London bank ATM did with my bank card after one failed PIN attempt.  Nice.

So, after finding my specific Mac OS X install disc, I did a boot to the disc, chose ‘Restore System from Backup’ from the Utilities menu, selected the backup volume and the Time Machine backup I wished to restore to (most recent).

It says 2 and a half hours in the picture above, but it turned out to only take about an hour and a half.  While this churned away, I got started on an all day patch extravaganza of our new HP Windows 7 laptop (for the new sales guy).

Step 4: Restore Boot Camp Partition with Winclone

I would have been done at this point, except that my Windows install uses Boot Camp.  I’ve been a fan of Winclone for a couple of years now, though it turns out they stopped development and decommissioned their website!  I understand someone has released an unofficial copy of Winclone 2.3, which is supposed to support Lion (I believe you can just edit the appropriate plist file to ignore the version check if you want to continue using Winclone 2.2).  See my side rant below.

To restore Windows, I opened the Boot Camp Assistant, told it I have the Mac OS X install disc, then chose Create or remove a Windows partition, and then set the partition size to 40 GB.  I did not format the drive, as Winclone will do this for me.

I’ve used Winclone before when increasing the size of my Boot Camp partition.  I simply blew away the partition with Boot Camp Assistant, increased the size, and then restored using Winclone.  I did the same thing here with my hard drive swap.  To perform the restore, you choose the restore tab, pick the backup file to restore to and choose the Boot Camp partition as the destination, unless you really want to repeat step 3 above.


Side Rant

I must have been snoozing or something, because I had no idea the developers of Winclone over at twocanoes.com were in trouble or were even considering abandoning Winclone, which they did sometime in 2010.  However sad that may be, why in the world did they just dump it?  Seems like it would have been just as easy for them to move it to source forge and turn it over to someone else.  I’m not aware of any program that replaces it, so it’s not like they gave up due to the competition.  So, my rant?  If you’re going to stop developing a free product that lots of people use, at least put it up on source forge and ask someone to take it over.  For me, moving forward, I figure it’s just as easy to simply dd the partition to an external drive.  That could even be scripted and run from cron.

Hmm, if only there were some site that would discuss how to do things like that, such as  backupcentral.com (this link points to an online version of the Linux and Windows Bare Metal Restore chapter I wrote in W. Curtis Preston’s book, Backup & Recovery).

Step 5: Verify Everything

It should be pretty obvious if the restore succeeded or not.  If things are all wonky and restoring to a different backup in Time Machine doesn’t work, then you will be stuck with a re-install of the OS, followed by restoring data from a backup (maybe the file restores will work from Time Machine or maybe you have another product grabbing files, like I do with CrashPlan).

For me, everything was working fine, except for a couple of things, one of which is minor, the other turned out to be a non-issue.  One was Microsoft Office and the other was my Parallels version of Windows.

Microsoft Office had two issues, a database index that it said it needed to rebuild, and a prompt requiring me to re-enter my product key.  Both of these were pretty minor, though still unexpected.

The other issue was Parallels.  I use Parallels to run my Boot Camp version of Windows as a virtual machine while running Mac OS.  However, after my recovery, there was no Parallels applications folder in my dock and the pvm file was missing to start-up the virtual machine.  There is an option within Parallels that says “Don’t backup with Time Machine” and I’m pretty sure I checked that way back when I set this up, since I’m using Winclone.  However, I’m not sure of that, and it’s not the default.  So, I simply went to CrashPlan to restore the pvm file, but discovered that it was 30+GB, which didn’t make sense, since it should just be a small file around 25 MB pointing to the Boot Camp location.  All I did was start-up Parallels, choose “New” and point it to the Boot Camp Partition, then I was back in business.

The nice thing about testing your restores with a hard drive swap is the built-in fallback should it have failed.  I could have gone back to my puny little 128 GB drive.  Good thing I didn’t have to do that!  Who lives with such small drives these days? ;)

→ Leave a comment

Posted in Backup Stuff

Tagged , , , , , , ,

Microsoft System Restore Failure Leaves Me Hanging

Posted on August 3, 2011 | Leave a comment

So, I’m minding my own business, just getting ready for another seminar, setting up our one and only Windows laptop to our projector when Power Point decides to crap out.  Reboots didn’t help and time was running short.  No problem, right?  Just roll back to a previous restore point and I should be fine.  Well, that didn’t turn out to be true.  I’m actually not that torqued that rolling back didn’t fix anything, I’m torqued that the rollback failed and the chose to wipe out all my other restore points!

I chose a restore point that was just before the last Microsoft update, thinking that may have been the culprit and knowing that everything worked fine the last time I used the laptop.  However, the system restore failed with an “unspecified error” and told me I should disable my antivirus and try again.  It then informed me it would have to undo the attempted restore.  Ok, sure, why not?  Here’s the fun part; the system restore got to the end of the rollback, failed and spit out an error that says something to the effect of “unable to rollback from system restore since the system restore did not complete correctly”.  That’s genius!  So, I have to rollback due to a failed restore, and simultaneously can’t rollback due to the same failed restore.  Oh, and just for good measure, we’ll destroy all the other restore points that were there so that you can’t use any of those or attempt to do this again.  Nice.

If you were wondering, the antivirus I run on this laptop is Microsoft Security Essentials, which apparently Microsoft System Restore doesn’t know how to interact with.

End of the world?  No, we proceeded with our broken PowerPoint and showed our video clips outside of PowerPoint and lived without answers to all of our audience response slides.  Once back home, I was able to uninstall and reinstall Office 2007 and get things working again (no, repair didn’t work).  I also had an image I could have used to perform a bare metal restore, though I failed to bring it along on my external drive.  It wouldn’t have mattered as I didn’t have an hour or two to wait for a bare metal restore.  I have, however, managed to get the audience response system(the only reason we need to run PowerPoint on a Windows computer) working on Windows running on Parallels on two of our Macs, so I have multiple redundant systems moving forward.

→ Leave a comment

Posted in Backup Stuff, Reed's Rant

Tagged , , , ,

When Web Apps Aren’t

Posted on June 29, 2011 | Leave a comment

I think it’s great that there are companies using web applications and not requiring me to install yet another application on my laptop.  One nice advantage of a web app is that it is a thin client, utilizing my existing browser,  allowing the work to be done on a server somewhere else and not on my computer.  Properly written, it also allows me access from any internet connected computer.  However, if your application only works in one particular browser, then I say phooey to you!

We have one application that only runs on Internet Explorer.  It bears mentioning that I am a Mac user, so I can’t just install Internet Explorer.  I do have Parallels installed to run any Microsoft apps that come my way, though it turns out the only app I’ve had to run on Windows so far is Internet Explorer!  I was expecting to have several applications that I would need to install in Windows, but no, just the web browser.  To make things worse, I updated Internet Explorer and then was unable to use the single application I have that depends on that browser and thus had to downgrade.  As for one of the advantages of a web app?  I hardly think that Windows 7 (since I can’t just install IE on Mac) counts as a thin client.  Even if I were still on Windows, I’d prefer to just install an app, rather than relying on a single browser and hoping that one of the many Microsoft OS patches doesn’t affect the usability of such a narrowly defined web application.

Ok, that’s it, I’ll step off my soapbox now and return you to your regularly scheduled programs (which can hopefully run in at least more than one browser).  And No Thank You to those of you about to offer me some cheese to go with my whine.

→ Leave a comment

Posted in Reed's Rant

Backing Up With CrashPlan

Posted on June 6, 2011 | 1 Comment

We moved into our new office in February of 2011.  One of the first things we did was to get our lab back up and running (well, after installing the new fridge to cool my soda).  Having heard good things about CrashPlan, we decided to give it a test run.  The short version of the story is: it’s easy to install, administer, restore from and even easier to migrate to new hardware than I originally anticipated.

Environment

Just a quick review of our environment. We are backing up 2 Macbook Pro laptops and 1 iMac using the free version of CrashPlan.  We are backing up to a Windows 7 box in our lab with a 16TB Drobo attached to it.  Yes, we could have backed up each laptop to the CrashPlan site and not had to point it to our own server and storage, but where’s the fun in that?  We may add that functionality in the future.

This is not intended as an in-depth review or detailed walk-through of the product.  I was actually impressed with the ease of migrating to a new server and decided to share my experience with the product.

Installation and Use

Installation on all 4 machines (the macs and the backup server) took only minutes and was a simple process.  The initial backup to our local server took the better part of a few days to complete, mostly because one of us actually has some 200+ GB of stuff on his 500 GB drive (I can “name fingers and point names”, but we’ll just say his name rhymes with Wurtis).

Restoring a file was as simple as walking through my filesystem in the CrashPlan GUI and selecting the file to restore, which restores the most recent copy to the Desktop by default, but it is easy to click and change which version and where to restore.  I even did a restore while the backup was still running.

I’ll note here that administration with this version is rather minimal, as it was designed for personal backups, CrashPlan Pro is really where you would go to manage backups for multiple computers.  With the version we have, I can see the status of all 3 laptop backups only if I log into CrashPlan on the backup server, and only get notifications for backups of my own laptop.

Migrating to New Hardware

Ok, so this would not have been a problem I would have to worry about if I had simply enabled the offsite feature and backed up to the CrashPlan site.  However, there would not have been as much to test or play with had we gone that route.

The problem was our backup server which started to flake out on us.  Since it is running on hardware that wasn’t exactly the newest and fastest when we bought it some 3 or 4 years ago, I didn’t even bother trying to spend much time troubleshooting and fixing. I just bought a new box from Fry’s.

Once I had gone through the 5,000 or so windows updates/patches, I installed CrashPlan on the new server, shutdown CrashPlan on the old server and moved the Drobo to the new server.  I was anticipating re-configuring each of the 3 clients to point to the new server and having them re-sync themselves.  I knew they wouldn’t need to start over on the backups since we essentially have the backups seeded on the Drobo.  I was pleasantly surprised to learn that it was even easier than that.  Once I started CrashPlan on the new server, and signed in with the same account as the old server, it showed me both computers the account was assigned to and asked me if I wanted it to take over (adopt) backups for the other computer.  Why yes I do, thank you very much!  That was all it took.  Backups resumed for all 3 clients with no additional configuration needed.

Summary

As I said, easy to install, easy to restore from and even easy to move to new hardware if you’re using the onsite option.

→ 1 Comment

Posted in Backup Stuff

Tagged ,