06. Backups Made Simple (and Safe) - No Excuses! Backup now 💿➡📀

May 1, 2021 5 min read technical

If you already know the the importance of backups - skip to the youtube demo at the end of the article - to find out how you can backup your data securely and rapidly, else read on !

Most people realize the importance of data backup when it is too late.

Let that sink in

Are you one of them?

Many of us have heard horror stories about a crashed computer, something we definitely don’t want to happen to us. Not while we don’t have the backup of all the many years worth of data from our desktops and laptops.

Interestingly many of us have at least few hundred GB sized USBs, external harddrives lying around and we still put off backing up to “later”.

This is one of those things - while we know how important it is, we kick the can down the road anyway.

There are 2 kinds of backups

  1. Full disk/volume backups - these let you restore your computer when your harddrive crashes
  2. Partial backup/sync - Backup of folders and files that you cannot (easily) live without. Documents, photos etc.

Now the first kind of backups are large (upwards of 200Gbs for a windows installation) but they provide the best data protection. They generally require a third party software.

And the second type -which we focus on in this blog post - A good partial backup/ sync process should be able to do following

  1. Copy new files from source to destination (external HDD/ network share)
  2. Delete files/ folders that were deleted from the source after the last backup
  3. Relay the updated folders and files to the destination
  4. Allow for excluding certain files and folders (developers - I am talking about node_modules, java targets, mvn local repo and even .git folders)
  5. Store a log of what it did during the backup run

Seems pretty straight forward right ? then why have you not been backing up regularly ? even with so much of additional storage at your disposal (external hard drives, USBs etc)

I don’t know about you - but here are my general findings

  1. Most backup software applications provide imaging and syncing capabilities but it’s really hard to trust them with all your personal data.
  2. Many do this in their proprietary format instead of sticking to standard formats
  3. Many of them are paid products and those that are free, lack the support
  4. Cloud backups (Onedrive, Dropbox etc) - costs add up pretty quickly
  5. Above points make people want to stick to standard windows copying for all the important folders and that is cumbersome and so slow !
  6. So then - few people do backups !

Anyways before I make you feel helpless about the situation, here’s what I have recently come up with after using different backup products and strategies; this fits my needs of important data backup.

  1. It is transparent
  2. It is simple and open source (I wrote it !)
  3. It’s less than 50 lines of code with decent documentation (this blog, the video at the end explains the code as well along with the demo)
  4. Uses standard windows / Mac / Linux commands - so nothing proprietary
  5. You can run it when you want or schedule for regular backups

So here it goes - and as always there is a demo at the bottom of this blog post

Windows 10 -

The script uses native robocopy command and powershell

$time_stamp=[datetime]::now.tostring("MM-dd-yyyy-HH-mm-ss")

$log_file_name=$time_stamp+"-backup.log"


foreach($line in Get-Content .\backup_conf.csv) {
    $elements =$line.Split(",")
    $source_dir=$elements[0]
    $dest_dir=$elements[1]
    $exclude=$elements[2]
 
    $command_and_options = "robocopy "+$source_dir+" "+$dest_dir+ " /mt:10 /mir /log+:"+"$log_file_name"

    if ("" -ne $exclude -and $null -ne $exclude){
        $exclude=$exclude.Replace("#", " ")
        $command_and_options = $command_and_options+" /xd "+$exclude
    }
    
    Write-Output "------------------------------------"
    Write-Output "Backing Up $source_dir"

    Invoke-Expression -Command $command_and_options
    
    if (!$?){
        exit $?
    }else{
        Write-Host "Completed Backing Up $source_dir"
    }
}

To Run it on a windows 10 machine

Check out the demo at the end or follow these steps

  1. Save this script as simple_backup.ps1
  2. Create a configuration file backup_conf.csv in the same folder with following format
    source_dir_path,destination_dir_path,exclude_dir1#exclude_dir_2
    you can also skip the exclude dir configuration
  3. Start-> open powershell -> browse to simple_backup.ps1 -> ./simple_backup.ps1

Linux / Mac -

The script uses standard bash and rsync

#!/bin/bash

time_stamp=`date "+%m-%d-%Y-%H-%M-%S"`

log_file_name=$time_stamp"-backup.log"

for conf in `cat ./backup_conf.csv`
do
        source_dir=$(echo "$conf" | cut -d "," -f1)
        dest_dir=$(echo "$conf" | cut -d "," -f2)
        exclude=$(echo "$conf" | cut -d "," -f3)

        echo "Backing up $source_dir" | tee -a $log_file_name
        if [[ $exclude == "" ]] || [[ -z $exclude ]]
        then
                rsync -Cavz --delete $source_dir $dest_dir 1>>$log_file_name 2>> $log_file_name
        else
                echo "excluding $exclude" | tee -a $log_file_name
                exclude_file="exclude_file.txt"
                >$exclude_file
                for dir in `echo $exclude | tr "#" "\n"`
                do
                        echo $dir  >> $exclude_file
                done
                rsync -Cavz --exclude-from $exclude_file --delete $source_dir $dest_dir 1>>$log_file_name 2>>$log_file_name
        fi

        if [[ $? == 0 ]]
        then
                echo "Done backing up $source_dir" | tee -a $log_file_name

        else
                echo "Error Backing up $source_dir" | tee -a $log_file_name
        fi
        echo "-------------------------------------" | tee -a $log_file_name

done

To Run it on a Linux / Mac Machine

Check out the demo at the end or follow these steps

  1. Save this script as simple_backup.sh
  2. Create a configuration file backup_conf.csv in the same directory with following format
    source_dir_path,destination_dir_path,exclude_dir1#exclude_dir_2
    you can also skip the exclude dir configuration
  3. Open shell -> browse to simple_backup.sh -> chmod +x ./simple_backup.sh -> ./simple_backup.sh
And here's the demo video

Demo - Use Your Own Backup Scripts | Simple, Safe and Fast Backups !

Please watch in full screen or on youtube directly



Thank you for reading through

-Nikhil

comments powered by Disqus