Backup hell: a visit to the crypt

Lovely people

Let's be serious for a moment. Perhaps I haven't been paying attention, but I don't have the feeling that the police forces around the world are very concerned about malicious software attacks. They seem happy to monitor social media for signs of 'hate speech' or whatever is currently on the hit list, but I can't remember hearing of anyone being collared for creating or distributing some form of 'malware'.

This is not the same as hacking into servers and computer systems, that's a different activity altogether, one that gets lots of publicity. No, we are talking about the act of disrupting computer systems with malware, whatever its flavour: keyloggers, DDoS agents and so on. The pests who do this cause much misery, but, despite that, never seem to get caught. Perhaps catching them might disrupt the 'antivirus' business that has grown up on the basis of this threat. Tut, tut, surely not! Such cynicism at this time of goodwill to all men!

The malignancy of malware went to a new level a couple years ago with the invention of 'ransomware'. This development depended on the payment anonymity offered by the new digital currencies, principally Bitcoin.

Ransomware encrypts certain classes of files on entire disks, making them inaccessible to the user, who is required to make a Bitcoin payment of some amount in order to be given the key to decrypt the data. Some recent versions will even steal any Bitcoins found on a computer, ironically leaving the victim with no means of paying.

The mugged user who does pay up can only hope that that payment is the end of the matter, not just the start of a blackmail auction; can only hope that the criminals are as good as their word and send a decryption key and don't just take the money and go silent; can only hope that the criminals did, in fact, encrypt the data and not just destroy it, never intending to recover it. In this game, the victim has no cards at all to play.

The companies selling antivirus software make hopeful noises about being able to decrypt poor quality encryption keys, but this is just bluster: the cost of individual analysis and cure is probably higher than the ransom demand. A recent piece in The Register attempted optimism, but ended in resignation.

Research last week from IBM's X-Force security team chatted to 600 business customers and found 70 percent of them had paid ransomware spreaders to get their data back. Over half paid $10,000 in ransom and one in five coughed up over $40,000 for the keys to their data.

Ransom payment was much less common among consumers, the same study found. Around half of the 1,000 people polled said they'd pay up to get their data back, but were very price sensitive about it. Barely a third said they'd pay more than $100 for the cure to the malware.

In July CERT-EU, along with local teams in Ireland, Luxembourg, and Slovenia and a host of smaller security companies, launched the No More Ransom project to combat this particular form of nastiness. Now more than 30 groups, including Intel, European law enforcement, larger security firms, and volunteers are trying to find a cure for the problem.

It's going to be a long, hard slog, if it's possible at all. But the more people pay, the bigger the problem will become.

The answer isn't difficult to implement – make frequent and regular backups. Traditionally that's been something your average Joe hasn't been very good at, but there's no excuse for businesses not having the right secure storage systems in place.

Business is business: according to IBM estimates, in the whole of the year 2015, $25 million was paid out to ransomware. In only the first quarter of 2016, $209 million. The first and easiest conclusion: never pay the ransom. Those who do are simply feeding the beast. Protect yourselves instead.

That backup thing

'Frequent and regular backups'. Well, yes, very nice. Non-specialists are sold shiny computers and operating systems, but no seller gives its buyers the fright of their lives by telling them what will sooner or later happen to their shiny new thing. There is a reason for this: when disaster strikes most people throw in the towel and buy a new shiny thing. Business is business. We really are being unseasonably cynical, aren't we?

Some vendors offer built-in backup solutions which save data to a separate partition of the same disk. It is hard to understand how supposedly IT specialists can offer such idiocy, because, of course, if the disk fails or is attacked by malware the backup disappears, too. The lovely specimens of humanity that create ransomware specifically target known backup file formats.

It would be refreshing to find a dealer selling a computer with a dedicated external backup disk, a recovery CD and a backup schedule already set up. An outlet that does not do this really does not care a jot about its customers.

And, of course, there is human nature. Be honest, when was the last time you thought about what you would do if your home burned down? How many times have you imagined what precisely you would do if your computer crashed or was locked up with ransomware. Every bit of data on your computer, emails, photos, music, texts – gone forever. You can't access the internet, get help, use your email. With an IMAP email account you may be able to get your emails back, but everything else would be gone.

Even Microsoft, who really should know better, doesn't choose to make much of the downside of computing. As evidence we note how pathetic and limited the built-in Windows Backup was, right up to and including Windows 7. Not only do computer and software manufacturers neglect to equip users with a robust configuration for backup, they fail to tell them how to structure and run backups.

As the Figures of Speech contribution to the happiness of humanity at this festive season, here is a description of a backup system for an individual user. There are many other possibilities, so this is not to be taken as the only solution. Of course, if you are part of a network the solution will be different and you should have specialists to do that.

Happy-go-lucky solution

If you can't really be bothered to read any of the advice in this posting on how to back up your computer efficiently then you should at least do the following as a very minimum. Not perfect, but better than nothing.

1. Get an external USB drive. The drive should have a capacity of about two – better, four – times the capacity of your computer storage. It doesn't need to be the fastest disk on the market.
2. Back your computer up at least once a week. The less frequently you perform this backup the more data you will lose in the case of a catastrophic crash.
3. Only plug in the external USB backup drive just before running a backup. After the backup, 'dismount' the external drive by using the 'Safely remove hardware option' in the notification area of the taskbar. Do not leave the drive plugged in. Do not just pull the connector out. You may often get away with doing this, but your backup disk is precious, well worth the few seconds it takes to disconnect it from the system properly. Run no unnecessary risks with it.
4. Use the Windows backup software, it's better than nothing. But there are a number of applications available that do the job much better. Macrium Reflect is the one I use, but there are plenty of others, AOMEI Backupper, for instance.
5. Make sure you have an up-to-date recovery CD or USB-stick. Windows will make this for you. A back up is no good – a complete waste of time, in fact – if you can't recover the data from it.
6. Print out on paper the section from Windows Help that tells you step-by-step how to recover a PC. Store it somewhere where you will remember it. If you don't do this you will one day find yourself staring at a dead computer with no idea of what to do next.
7. Hope for the best.

That's it. Stop reading here and go and buy a suitable disk and give your computer at least this basic protection. If nasty things happen, your maximum loss is one week's worth of data, or whatever your last backup interval was. If the data from that interval is precious to you, then our quick and dirty solution is not suitable – you need to read the following section of this piece.

Backup efficiency

Just making full backups at certain intervals is better than nothing, but it is tedious if the intervals are short and it leaves more data unprotected the longer the intervals are. How can we protect the important work we do in the intervals between backups?

With a little optimisation effort in the beginning we can not only do this but we can make the whole process much more efficient and secure. The first step is to design our data storage so that it makes backing it up as simple, efficient and as foolproof as possible.

Separate data and system

Separating data and system has been a mantra since the earliest days of computing. There are many good reasons for this principle, but the main one is easy to grasp. If you mix up your data (documents, photos, music) with system files (programs etc.) the confusion makes it difficult to implement a clean backup strategy. It is usually not possible to make simple copies of system files (access rights, files in use etc.), the only way to do this is by using special system backup software. If you keep your data together in either its own folder structure or – best of all – on a separate disk, this will make your life much simpler in the long run.

Well, the principle is good, but the implementation is not so straightforward, particularly under Windows. For example, many Windows applications will try to force you into using the 'Documents and Settings' structure in your user account. Wherever reasonably possible you should try to avoid this. Badly programmed applications will not even give you the choice.

As more complexity has been added over the years, 'Documents and Settings' has become a nightmare maze of data, configuration files, virtual folders, linked folders and so on. It should now be considered to be part of the system and not a suitable storage location for your data.

From the execrable 'Windows Vista' onwards Microsoft also introduced the confusing idiocy of virtual 'Libraries', which made it even more difficult to rationalise the storage of data.

If you don't have a separate online disk available for your data, take some time to add a folder at the root of your C: drive called, say, 'Data' and create a structure underneath it that corresponds to the type of data you have and how you work with it. If you put your data where it logically belongs then finding it and backing it up will become much easier. But the separation of system and data is not the only thing we need to consider.

Separate volatile and inert data.

Let us say that I make a folder called 'Official Docs' and put there scans and copies of the documents that the modern state requires of me, poor, ignorant grain of corn between the millstones that I am. Copies of passports, residence permits, birth certificates etc. could be very useful if I lose the physical originals: not replacements, but a help in obtaining replacements. How often does this information change? Hardly ever: my birth certificate is already there, it just needs my death certificate and the data store is complete – at last I can rest in peace!

There is no point backing such data up every ten minutes – it just wastes time and space. Let's call it inert data.

In contrast, let us say that I make a folder called 'Web sources' and put there all the documents I write for websites, plus related images, notes and so on. The small portion of this stuff that I am working on at any particular moment changes very rapidly. Anyone who writes and has lost so much as a paragraph for whatever reason knows how particularly annoying it is to have to rewrite the lost section afresh.

This we can call volatile data. Of course, there may be some relatively inert data alongside it: once a piece is written and used it will stay unchanged and becomes inert. But we want to be able to backup up data without having to think about this too much and so we can accept the slight inefficiency of the mixture. There are ways of backing up that can reduce this problem considerably, but our goal is to keep the volatile data in a lightweight, compact structure that can be copied conveniently in seconds.

Since volatile data is being accessed and worked on intensively, there is also more risk of bad things happening to it: files may have been accidentally deleted or become corrupted through some software failure or we may have messed them up to such an extent that we want to return to an earlier version. A sensible system for the frequent backup of volatile data will be able to help in these cases.

Minimise nuisance

No one ever wants to back up. We want to concentrate on our current work. We have to take into account the trade-off between time and data volume. Disks are high capacity these days and making complete backups is tedious. It is especially tedious when we realise that the vast majority of the data has not changed recently. If this were the only backup we did, we would soon lose interest and the backup intervals would start to drift apart.

We therefore need a scale of acceptable nuisance. Backing up important volatile data should be as fast and as frequent as possible. Let's call these hot backups, whereas full backups should be done at regular intervals in order to preserve the full system and all data. The same inert data will be backed up again and again, but this redundancy has its value.

Implementing an efficient backup system

Hot backups, online

We need a completely separate and permanently online disk for our 'hot backups'. If you have a desktop with a big enough box you can keep the disk there, if you have a laptop with space for a second drive you would put it there, otherwise an external USB or SATA drive will be required. Yes, I know, I know, one more thing to carry around…

I use a program called Second Copy to copy volatile data from my data disk to the backup disk. There are plenty of other programs available that do the job. I have got into the habit of running a backup profile on volatile data every ten minutes or so during periods when I am working on data. On my system the copy process typically takes no more than a second. I don't schedule this hot backup, because I know best when I need to do it. Backing up automatically every five minutes might even overwrite good data with bad.

The copy program only copies data that has changed, meaning that it is extremely quick. It also saves a certain number of previous versions of a changed file, a useful feature when things go wrong. People who can use the Windows command line and know about 'Robocopy' can write a batch file that will do much the same thing.

I also use the disk as a backup target for a number of other programs. For example, I have an image management program with its own backup procedure which I have set up to write to the hot backup disk. This is a useful protection against data corruption or disk crashes. In this way hot backups secure all or nearly all of my volatile data in case of a disk failure, accidental deletion or scrambling of a file.

Because it will be accessed very frequently this disk has to be online. We don't want to be faced with the tedium of mounting an offline disk, running the copy and then dismounting again. But this is a weakness. Ransomware encrypts ALL online disks, which means that the hot backup disk will be lost, too. ^[1]

Hot backups, offline

In order to protect against the threat posed by ransomware I run the hot backup onto a USB-stick two or three times during the day, depending on my workrate. This takes a minute or so: wait for the USB-stick to mount, run the copy (a few seconds), 'eject' the USB-stick. In effect we make a risk assessment about ransomware: the risk of data loss against the possibility of its occurrence traded off against the inconvenience of using a USB-stick for a minute a few times a day. Your choice.

Full backups, offline

We will need a further disk to hold the full backups of all the data on the computer together with a backup program to carry out this task for us. I use Macrium Reflect (the free version is quite adequate, but you might consider the paid version for its ability to back up parts of file systems, i.e. your volatile data), but there are plenty of other programs available. The Windows backup system – I don't know how it is in Windows 10 – has many drawbacks such as extreme slowness and very shaky recovery procedures, so I personally would never trust Microsoft to back up and recover my data.^[2]

The full backup disk should be big enough to take multiple complete backups. For example, if your main disk is 500GB and it is half full, even with data compression during the backup you should consider at least 1TB (i.e. 1,000GB) for the backup disk.

Why do you need to keep multiple backups? Because your latest backup may turn out to be corrupt in some way, or even contain the seeds of the malware that you are now trying to escape.

Don't use sophisticated backup methods with so-called differential or incremental backup archives. These can save space and time for huge quantities of data on large systems that are backed up frequently, but are intrinsically less robust than single files. Disk space is inexpensive at the moment and the need for the management of backup sets adds one more layer of complexity to the process that really only system administrators need to think about.

You don't need to backup the hot backup disk. The main backup contains all the source data that has been put onto this disk.

If you have managed to separate system and data onto two disks back them up separately. If you lose one of the two disks you can recover that disk specifically. Each profile can be run separately, which means they are individually faster and allow you to choose what to run and what to restore.

The full backup drive should not be permanently online. If you are attacked by ransomware your backups will also be useless. If you have a big desktop case you can keep the drive in a drive slot of the case but not pushed home. When you want to make a backup, push it in, give it time to mount, run the backup, dismount it gracefully and pull it just out of the connector. Using internal SATA channels for data transfer will be quicker than external USB ports and cables.

You could run these backups once a week. You can also run them ahead of making major changes to the computer, then if things go wrong there will always be a way back.

On my system a full backup takes about two hours for both disks. During those two hours online you are vulnerable to a malware attack, but if you don't do any surfing around and don't read emails offering you pictures of cute cats or the Kardashian family you should be safe. Despite the assurance that you can 'carry on working' during the back up, wherever possible you should only back up 'quiet' systems.

If disaster strikes, in the absolute worst case you will be able to recover your basic system to its state of a week ago. Your volatile data can be recovered either from the hot backup disk, if it is intact, or the USB stick. You may be slightly unlucky: there may be a few bits and pieces outside the hot backup that you have added or changed after the full backup that will be lost, but you make your own luck by ensuring that important things that cannot easily be recovered are protected until the next full backup. In comparison with the value of what you have saved, such unlucky losses are trivial.

It's up to you to decide how to organize your data to keep it safe. You just have to think, at any particularly moment, what you would lose and could not reasonably recover and what trade-off in time and convenience you are prepared to accept for better security.

Extra precautions

Precautions for the apocalypse

If you are a worrier like me, you should get a separate disaster disk, at least as big as the main backup disk, and make a copy (using 'Robocopy', not the Windows Explorer copy function) of the latest backup files from your main backup disk.

I update my disaster disk about once every month, or whenever I remember. You need to have both the backup disk and the disaster disk online, so during the two hour copy process don't mess about online.

Keep the disaster disk somewhere where it will not be affected by fire or other misfortune. Again there is a trade-off between convenience and threat: if you keep the drive in a bank-vault how often will it get updated? If you keep it in the cellar, what happens when that floods or the firemen fill it with water? In case of asteroid impact, who cares anymore?

Precautions for the sensible

A serious computer crash is not repaired with a snap of the fingers. You may need to purchase another disk, or even another computer. Until you get it all up and working your digital life is in tatters. When a disaster occurs you will be extremely unpleasantly surprised at how often you need to access the internet or to send and receive emails in order to fix your system and to enable you to keep in touch with people until you do.

For this reason I also maintain a USB-stick that contains all my digital life: passwords, browser and favourites, useful programs and diagnostic tools, various documents and data that I might need if I have to access my digital life from someone else's computer.

All this sensitive data is kept on a hardware encrypted USB-stick. If I lose it, it doesn't matter much. I keep an offline copy of the data in case I ever need to rebuild it. I keep it in the car, so it is almost always with me. When I am away from my home system I have everything I need for my digital life on someone else's computer or for use in restoring my own system.

Notes

1. ^ Computer specialists reading this are now saying to themselves: yes, but you could just mount and unmount this volume with 'mountvol'. Indeed I could, but a) this is one more tedious step and b) I am not entirely sure that an unmounted volume that is still physically present will be invisible to ransomware. That is why we take the next step, in which the drive is physically removed.
2. ^ A few years ago Windows Backup refused to recover a disk for me because it did not like the configuration of the replacement disk. Every recovery attempt took about half an hour of loading drivers and selecting options before issuing a completely obscure message of the form 'Disk recovery cannot be completed'. Never again.

Update 03.02.2017

The headline and straplines tell you the tale.

Town hall has two years' worth of documents deleted by virus that demanded £3,000 ransom after clerk mistakenly opened virus-laden email

• Tiverton Town Council lost two years' worth of files after email virus attack.
• Message claimed to be from parcel delivery firm but was actually ransomware.
• It demanded £3,000 in exchange for all the documents to be unlocked.
• Council set to spend months re-uploading the files into its IT system.

The conclusions are depressing:

1. The Town Clerk is so stupid that he runs a computer system for two years without making a backup.
2. The Town Clerk is so extremely stupid that he allows that stupidity to become public.
3. He now says: 'Our anti-virus wasn't good enough for this one. It is a warning to other people and it I think our security for this type of thing needs re-thinking', thus revealing publicly that he doesn't get it at all, even after the fact. He has learned nothing. Unfortunately for the taxpayers of Tiverton, the cost of months of document scanning will not come out of his pocket.

Let's be kind: he should consider getting a better IT support provider, who would tell him to stop messing about with antivirus software and institute a robust backup system as described in our article (with adaptations for a multi-seat system).