How often do you test your backups ?

[phenomlab]

How often do you test your backups ? Whether it’s weekly, monthly, quarterly, or an even longer duration than that (if that is the case, stop reading this and test your backups now), every business and personal user needs the assurance that they can recover either entire systems, or an individual file in the event that a need arises. Whilst you may be thinking “yeah, obviously”, you’d be surprised at how often this critical process is the victim of oversight on the part of the operator, or forgotten completely in the rush to have a system ready by a particular deadline. We’ve all missed things on server builds like antivirus, or the latest patches, but no backup processes ?

No system should ever be considered ready for release to production (or even test in some cases) if there is no backup strategy in place. This is nothing more than unprecedented risk to the business - particularly if the service is considered critical to any particular function. Can you imagine having to explain to senior management or stakeholders that the system everyone has been using for months has just crashed, and you’ve just suddenly remembered that there isn’t a single backup ? You may as well start updating your CV or resume now. I’ve never had to explain to senior management that there is no backup, and I have no desire to start doing so now.

A backup strategy isn’t difficult to define, but even with one in place, there is still a risk that this may not function as expected in the event of needing to recover a system or individual file. Backups should be regularly tested to ensure both consistency and integrity, with the restore process being validated afterwards - in short, you are attesting to the fact that you are in a position to recover either an entire server image, or a single file (dependent on the need). Failure to comply with this simple task could see you being thrown to the lions unnecessarily. Backups themselves are not difficult to undertake, and should be running on a daily schedule at minimum. The actual strategy varies heavily between use cases, with some taking differential or incremental backups during the business week, and then executing a full backup over a weekend. This type of backup is still very popular as it has the huge benefit of backing up only files and folders that have changed - the downside is that in the event of a disaster, you’d need the last full backup to be restored first before you can play back each incremental. Another more common approach is to use delta based backups that leverage bits technology to make the most of compression in order to reduce storage requirements and costs. Usage of delta backups typically requires a concept called “safesets”. It’s more commonplace in today’s technology sphere to use an off-site vaulting mechanism where you create an initial “seed” of a backup target, then ship only the changes to that same system on a daily basis.

This type of backup is also referred to as a snapshot. In the case of (for example) Amazon Web Services, the underlying technology is hypervisor based. This means that an initial image can be made of the machine in question, then a daily / hourly snapshot performed which captures the changes since the last snapshot was taken. In the event of a restore or disaster recovery, the initial image is recovered first, with the snapshots being “layered on top” to provide the necessary recovery point - this itself really depends on your backup strategy. One thing to bear in mind with off-site vaulting is that the costs can easily spiral out of control with penalty charges being applied for exceeding your quota limit. Storage itself is relatively cheap and easy to acquire, but the backup process required to support it can often attract significant and unprecedented cost to any user. For this reason, the backup strategy should be well defined, with best practices playing a major role in determining both the strategy and the overall concept. It’s important to note at this point that although there are numerous product offerings that include cheaper storage options, these quite often become the Achilles heel for those managing systems - the storage is cheap, so you get to save a fortune. However, the RTO (Recovery Time Objective) shoots through the roof - that cheaper storage is often magnetic - meaning it could reside on a tape somewhere. The restore times are not just limited to the speed of the media. Tapes typically require a catalog to be built first before backup software can be in the required position to recover the data in question. Years ago, I was performing backups and recovery testing on DLT tapes via a single loader and low grade SCSI card. A restore of an Exchange database could take well over 24 hours to recover based on this aged technology, and a slow network made this even worse.

Thankfully, we are not bound by slow hardware in today’s modern technology era. However, if your backups remain untested, how do you know that they will serve the purpose they are designed for ? The short answer is that you don’t. Each backup taken should be subject to analysis in terms of the log files generated during the job - in other words, the backup log should be checked on a daily basis to determine whether it was successful or not, and any issues noted be rectified as soon as possible. With ransomware wreaking havoc across the world, it also makes perfect sense to perform sanity checks on servers and their associated backups to ensure that you are not actually taking copies of files and folders that have in fact been encrypted.

Hit with ransomware ? No problem - we’ll restore from backup

……until you discover that your backups are also full of encrypted content.

Obviously not a great situation to find yourself in. For this reason, it’s important to invest in decent anti malware endpoint tools and file monitoring capabilities to prevent this from happening in the first place. It’s also very important to frequently test your ability to recover files from backup.

The bottom line - It’s not possible to say with 100% certainty that your backups are in full working order unless they have actually been tested.

[gotwf]

@phenomlab Yes. But I do not backup stuff as religiously as I should. Cobbler’s children’s shoes kind of deal. Kind of tough to get motivated when only dealing w/a few, rather than thousands, of boxes. Plus, I kind of like to keep my fingers into the cli - cuz if you don’t use it, you loose it.

That, and being retired, I no longer have unlimited access to the resources I might otherwise like to have, provided someone else is picking up the tab. :emoji: :emoji: :emoju:

OTOH, things are also lots simpler now.

P.S.; Yeah, I have a somewhat atypical sense of humor…

[JAC]

@gotwf said in Do you actually test your backups?:

Yes. But I do not backup stuff as religiously as I should.

You and me both then as @phenomlab has had to pick up the pieces so to speak a few times now I really need to make sure I keep everything backed up. I think Mark’s patience may run out next time, I’m like a cat with nine lives and I’ve already lost a few .

[phenomlab]

@jac @gotwf provided you have a sensible approach to backups, the cost needn’t spiral out of control - and, neither should the complexity. Most general consumers of data have large amounts of files that are typically static in nature - such as photos etc.

Clearly, these won’t be changing anytime soon (unless you’re into image editing) so you could arguably leverage a long term archiving solution for that data. This would keep the cost down to a minimum, and then you’d only need golden copies and one duplicate set just in case - and in most cases, you never access the golden copies as they are literally the last bastion if anything goes wrong.

Where several people fall in their own swords is to attempt to reduce costs further by keeping the backup of their data in the same place as the origin. Sure, you can argue that it’s on a removable hard disk etc, so if your pc crashes, and you lost the disk, you still have that data - great.

But what if you had damage caused by flooding or fire, or had your pc / laptop and the external disk stolen…

The correct strategy here is to keep your backups apart from the origin. In most cases, storage in a secure location (off site) or in a cloud based environment is generally the way to go. Storage is cheap these days, but the backup of that same storage can often work out expensive, which is why it’s always a good idea to shop around for the best deals.

[gotwf]

@phenomlab said in Do you actually test your backups?:

The correct strategy here is to keep your backups apart from the origin. In most cases, storage in a secure location (off site) or in a cloud based environment is generally the way to go. Storage is cheap these days, but the backup of that same storage can often work out expensive, which is why it’s always a good idea to shop around for the best deals.

USB drives in the TB’s are pretty reasonable these days. Some even come w/built in mirroring. If something like this would suit capacity needs, then I think I’d prefer to use them like large tape drives of old, Keep rotating on some schedule. Then keep them in a safe deposit box. Maybe encrypt. Would not want to trust cloud providers with all the eggs.

I have stuff mirrored on multiple boxes but if my house burns down I am admittedly screwed.

[phenomlab]

@gotwf said in Do you actually test your backups?:

I have stuff mirrored on multiple boxes but if my house burns down I am admittedly screwed.

Well, hopefully, that never happens !!