Skip to content

backup

(Written by Paul Cobbaut, https://github.com/paulcobbaut/, with contributions by: Alex M. Schapelle, https://github.com/zero-pytagoras/)

About tape devices

Don\'t forget that the name of a device strictly speaking has no meaning since the kernel will use the major and minor number to find the hardware! See the man page of mknod and the devices.txt file in the Linux kernel source for more info.

SCSI tapes

On the official Linux device list (http://www.lanana.org/docs/device-list/) we find the names for SCSI tapes (major 9 char). SCSI tape devices are located underneath /dev/st and are numbered starting with 0 for the first tape device.

/dev/st0   First tape device
/dev/st1   Second tape device
/dev/st2   Third tape device

To prevent automatic rewinding of tapes, prefix them with the letter n.

/dev/nst0   First no rewind tape device
/dev/nst1   Second no rewind tape device
/dev/nst2   Third no rewind tape device

By default, SCSI tapes on Linux will use the highest hardware compression that is supported by the tape device. To lower the compression level, append one of the letters l (low), m (medium) or a (auto) to the tape name.

/dev/st0l   First low compression tape device
/dev/st0m   First medium compression tape device
/dev/nst2m  Third no rewind medium compression tape device

IDE tapes

On the official Linux device list (http://www.lanana.org/docs/device-list/) we find the names for IDE tapes (major 37 char). IDE tape devices are located underneath /dev/ht and are numbered starting with 0 for the first tape device. No rewind and compression is similar to SCSI tapes.

/dev/ht0   First IDE tape device
/dev/nht0  Second no rewind IDE tape device
/dev/ht0m  First medium compression IDE tape device

mt

To manage your tapes, use mt (Magnetic Tape). Some examples.

To receive information about the status of the tape.

mt -f /dev/st0 status

To rewind a tape...

mt -f /dev/st0 rewind

To rewind and eject a tape...

mt -f /dev/st0 eject

To erase a tape...

mt -f /dev/st0 erase

Compression

It can be beneficial to compress files before backup. The two most popular tools for compression of regular files on Linux are gzip/gunzip and bzip2/bunzip2. Below you can see gzip in action, notice that it adds the .gz extension to the file.

student@linux:~/test$ ls -l allfiles.tx*
-rw-rw-r--  1 paul paul 8813553 Feb 27 05:38 allfiles.txt
student@linux:~/test$ gzip allfiles.txt 
student@linux:~/test$ ls -l allfiles.tx*
-rw-rw-r--  1 paul paul 931863 Feb 27 05:38 allfiles.txt.gz
student@linux:~/test$ gunzip allfiles.txt.gz 
student@linux:~/test$ ls -l allfiles.tx*
-rw-rw-r--  1 paul paul 8813553 Feb 27 05:38 allfiles.txt
student@linux:~/test$

In general, gzip is much faster than bzip2, but the latter one compresses a lot better. Let us compare the two.

student@linux:~/test$ cp allfiles.txt bllfiles.txt 
student@linux:~/test$ time gzip allfiles.txt

real    0m0.050s
user    0m0.041s
sys     0m0.009s
student@linux:~/test$ time bzip2 bllfiles.txt

real    0m5.968s
user    0m5.794s
sys     0m0.076s
student@linux:~/test$ ls -l ?llfiles.tx*
-rw-rw-r--  1 paul paul 931863 Feb 27 05:38 allfiles.txt.gz
-rw-rw-r--  1 paul paul 708871 May 12 10:52 bllfiles.txt.bz2
student@linux:~/test$

tar

The tar utility gets its name from Tape ARchive. This tool will receive and send files to a destination (typically a tape or a regular file). The c option is used to create a tar archive (or tarfile), the f option to name/create the tarfile. The example below takes a backup of /etc into the file /backup/etc.tar .

root@linux:~# tar cf /backup/etc.tar /etc
root@linux:~# ls -l /backup/etc.tar 
-rw-r--r--  1 root root 47800320 May 12 11:47 /backup/etc.tar
root@linux:~#

Compression can be achieved without pipes since tar uses the z flag to compress with gzip, and the j flag to compress with bzip2.

root@linux:~# tar czf /backup/etc.tar.gz /etc
root@linux:~# tar cjf /backup/etc.tar.bz2 /etc
root@linux:~# ls -l /backup/etc.ta*
-rw-r--r--  1 root root 47800320 May 12 11:47 /backup/etc.tar
-rw-r--r--  1 root root  6077340 May 12 11:48 /backup/etc.tar.bz2
-rw-r--r--  1 root root  8496607 May 12 11:47 /backup/etc.tar.gz
root@linux:~#

The t option is used to list the contents of a tar file. Verbose mode is enabled with v (also useful when you want to see the files being archived during archiving).

root@linux:~# tar tvf /backup/etc.tar
drwxr-xr-x root/root         0 2007-05-12 09:38:21 etc/
-rw-r--r-- root/root      2657 2004-09-27 10:15:03 etc/warnquota.conf
-rw-r--r-- root/root     13136 2006-11-03 17:34:50 etc/mime.types
drwxr-xr-x root/root         0 2004-11-03 13:35:50 etc/sound/
...

To list a specific file in a tar archive, use the t option, added with the filename (without leading /).

root@linux:~# tar tvf /backup/etc.tar etc/resolv.conf
-rw-r--r-- root/root        77 2007-05-12 08:31:32 etc/resolv.conf
root@linux:~#

Use the x flag to restore a tar archive, or a single file from the archive. Remember that by default tar will restore the file in the current directory.

root@linux:~# tar xvf /backup/etc.tar etc/resolv.conf
etc/resolv.conf
root@linux:~# ls -l /etc/resolv.conf
-rw-r--r--  2 root root 40 May 12 12:05 /etc/resolv.conf
root@linux:~# ls -l etc/resolv.conf
-rw-r--r--  1 root root 77 May 12 08:31 etc/resolv.conf
root@linux:~#

You can preserve file permissions with the p flag. And you can exclude directories or file with --exclude.

root ~# tar cpzf /backup/etc_with_perms.tgz /etc 
root ~# tar cpzf /backup/etc_no_sysconf.tgz /etc --exclude /etc/sysconfig
root ~# ls -l /backup/etc_*
-rw-r--r--  1 root root 8434293 May 12 12:48 /backup/etc_no_sysconf.tgz
-rw-r--r--  1 root root 8496591 May 12 12:48 /backup/etc_with_perms.tgz
root ~#

You can also create a text file with names of files and directories to archive, and then supply this file to tar with the -T flag.

root@linux:~# find /etc -name *.conf > files_to_archive.txt
root@linux:~# find /home -name *.pdf >> files_to_archive.txt
root@linux:~# tar cpzf /backup/backup.tgz -T files_to_archive.txt

The tar utility can receive filenames from the find command, with the help of xargs.

find /etc -type f -name "*.conf" | xargs tar czf /backup/confs.tar.gz

You can also use tar to copy a directory, this is more efficient than using cp -r.

(cd /etc; tar -cf - . ) | (cd /backup/copy_of_etc/; tar -xpf - )

Another example of tar, this copies a directory securely over the network.

(cd /etc;tar -cf - . )|(ssh user@srv 'cd /backup/cp_of_etc/; tar -xf - ')

tar can be used together with gzip and copy a file to a remote server through ssh

cat backup.tar | gzip | ssh bashuser@192.168.1.105 "cat - > backup.tgz"

Compress the tar backup when it is on the network, but leave it uncompressed at the destination.

cat backup.tar | gzip | ssh user@192.168.1.105 "gunzip|cat - > backup.tar"

Same as the previous, but let ssh handle the compression

cat backup.tar | ssh -C bashuser@192.168.1.105 "cat - > backup.tar"

Backup Types

Linux uses multilevel incremental backups using distinct levels. A full backup is a backup at level 0. A higher level x backup will include all changes since the last level x-1 backup.

Suppose you take a full backup on Monday (level 0) and a level 1 backup on Tuesday, then the Tuesday backup will contain all changes since Monday. Taking a level 2 on Wednesday will contain all changes since Tuesday (the last level 2-1). A level 3 backup on Thursday will contain all changes since Wednesday (the last level 3-1). Another level 3 on Friday will also contain all changes since Wednesday. A level 2 backup on Saturday would take all changes since the last level 1 from Tuesday.

dump and restore

While dump is similar to tar, it is also very different because it looks at the file system. Where tar receives a lists of files to backup, dump will find files to backup by itself by examining ext2. Files found by dump will be copied to a tape or regular file. In case the target is not big enough to hold the dump (end-of-media), it is broken into multiple volumes.

Restoring files that were backed up with dump is done with the restore command. In the example below we take a full level 0 backup of two partitions to a SCSI tape. The no rewind is mandatory to put the volumes behind each other on the tape.

dump 0f /dev/nst0 /boot
dump 0f /dev/nst0 /

Listing files in a dump archive is done with dump -t, and you can compare files with dump -C.

You can omit files from a dump by changing the dump attribute with the chattr command. The d attribute on ext will tell dump to skip the file, even during a full backup. In the following example, /etc/hosts is excluded from dump archives.

chattr +d /etc/hosts

To restore the complete file system with restore, use the -r option. This can be useful to change the size or block size of a file system. You should have a clean file system mounted and cd\'d into it. Like this example shows.

mke2fs /dev/hda3
mount /dev/hda3 /mnt/data
cd /mnt/data
restore rf /dev/nst0

To extract only one file or directory from a dump, use the -x option.

restore -xf /dev/st0 /etc

cpio

Different from tar and dump is cpio (Copy Input and Output). It can be used to receive filenames, but copies the actual files. This makes it an easy companion with find! Some examples below.

find sends filenames to cpio, which puts the files in an archive.

find /etc -depth -print | cpio -oaV -O archive.cpio

The same, but compressed with gzip

find /etc -depth -print | cpio -oaV | gzip -c > archive.cpio.gz

Now pipe it through ssh (backup files to a compressed file on another machine)

find /etc -depth -print|cpio -oaV|gzip -c|ssh server "cat - > etc.cpio.gz"

find sends filenames to cpio | cpio sends files to ssh | ssh sends files to cpio \'cpio extracts files\'

find /etc -depth -print | cpio -oaV | ssh user@host 'cpio -imVd'

the same but reversed: copy a dir from the remote host to the local machine

ssh user@host "find path -depth -print | cpio -oaV" | cpio -imVd

dd

About dd

Some people use dd to create backups. This can be very powerful, but dd backups can only be restored to very similar partitions or devices. There are however a lot of useful things possible with dd. Some examples.

Create a CDROM image

The easiest way to create a .ISO file from any CD. The if switch means Input File, of is the Output File. Any good tool can burn a copy of the CD with this .ISO file.

dd if=/dev/cdrom of=/path/to/cdrom.ISO

Create a floppy image

A little outdated maybe, but just in case : make an image file from a 1.44MB floppy. Blocksize is defined by bs, and count contains the number of blocks to copy.

dd if=/dev/floppy of=/path/to/floppy.img bs=1024 count=1440

Copy the master boot record

Use dd to copy the MBR (Master Boot Record) of hard disk /dev/hda to a file.

dd if=/dev/hda of=/MBR.img bs=512 count=1

Copy files

This example shows how dd can copy files. Copy the file summer.txt to copy_of_summer.txt .

dd if=~/summer.txt of=~/copy_of_summer.txt

Image disks or partitions

And who needs ghost when dd can create a (compressed) image of a partition.

dd if=/dev/hdb2 of=/image_of_hdb2.IMG
dd if=/dev/hdb2 | gzip > /image_of_hdb2.IMG.gz

Create files of a certain size

dd can be used to create a file of any size. The first example creates a one MEBIbyte file, the second a one MEGAbyte file.

dd if=/dev/zero of=file1MB count=1024 bs=1024
dd if=/dev/zero of=file1MB count=1000 bs=1024

CDROM server example

And there are of course endless combinations with ssh and bzip2. This example puts a bzip2 backup of a cdrom on a remote server.

dd if=/dev/cdrom |bzip2|ssh user@host "cat - > /backups/cd/cdrom.iso.bz2"

split

The split command is useful to split files into smaller files. This can be useful to fit the file onto multiple instances of a medium too small to contain the complete file. In the example below, a file of size 5000 bytes is split into three smaller files, with maximum 2000 bytes each.

student@linux:~/test$ ls -l
total 8
-rw-r--r-- 1 paul paul 5000 2007-09-09 20:46 bigfile1
student@linux:~/test$ split -b 2000 bigfile1 splitfile.
student@linux:~/test$ ls -l
total 20
-rw-r--r-- 1 paul paul 5000 2007-09-09 20:46 bigfile1
-rw-r--r-- 1 paul paul 2000 2007-09-09 20:47 splitfile.aa
-rw-r--r-- 1 paul paul 2000 2007-09-09 20:47 splitfile.ab
-rw-r--r-- 1 paul paul 1000 2007-09-09 20:47 splitfile.ac

practice: backup

!! Careful with tar options and the position of the backup file, mistakes can destroy your system!!

1. Create a directory (or partition if you like) for backups. Link (or mount) it under /mnt/backup.

2a. Use tar to backup /etc in /mnt/backup/etc_date.tgz, the backup must be gzipped. (Replace date with the current date)

2b. Use tar to backup /bin to /mnt/backup/bin_date.tar.bz2, the backup must be bzip2\'d.

2c. Choose a file in /etc and /bin and verify with tar that the file is indeed backed up.

2d. Extract those two files to your home directory.

3a. Create a backup directory for your neighbour, make it accessible under /mnt/neighbourName

3b. Combine ssh and tar to put a backup of your /boot on your neighbours computer in /mnt/YourName

4a. Combine find and cpio to create a cpio archive of /etc.

4b. Choose a file in /etc and restore it from the cpio archive into your home directory.

5. Use dd and ssh to put a backup of the master boot record on your neighbours computer.

6. (On the real computer) Create and mount an ISO image of the ubuntu cdrom.

7. Combine dd and gzip to create a \'ghost\' image of one of your partitions on another partition.

8. Use dd to create a five megabyte file in \~/testsplit and name it biggest. Then split this file in smaller two megabyte parts.

mkdir testsplit

dd if=/dev/zero of=~/testsplit/biggest count=5000 bs=1024

split -b 2000000 biggest parts