Backup: Difference between revisions

From 太極
Jump to navigation Jump to search
Line 107: Line 107:


== rsync with progress bar ==
== rsync with progress bar ==
Use '''--progress''' option. The speed is in MB (not Mb). I got about 16MB at home by using Ethernet.
Use '''--progress''' or '''-P''' option. The speed is in MB (not Mb). I got about 16MB at home by using Ethernet.
<pre>
<pre>
rsync -avz --progress file1 file2
rsync -avzP file1 file2
</pre>
</pre>
The 'rsync' command works on transferring files local to local too.
The 'rsync' command works on transferring files local to local too.

Revision as of 11:37, 19 October 2023

rsync

Dry run

rsync -avn master.bioconductor.org::release
rsync -n --dry-run master.bioconductor.org::release

Install

When using rsync to transfer data remotely, it must be installed on both the source and the destination machine. LibreElec does not include it by default.

I observe that "rsync" and "sshd" on the remote machine were using CPU when I copy from the remote machine to local.

Wildcard/glob

quoting is so important. Bash will replace it with a list of files before rsync is ever called. 

# Not working
rsync -avz Domain:Dir/myfiles* .

# Correct
rsync -avz Domain:"Dir/myfiles*" .
# OR 
scp -p Domain:Dir/myfiles* .

Copy large file

If we need to copy large file (say > 4GB), we shall

  1. format USB drive to NTFS (exFat seems not work)
  2. Run rsync --progress source dest
  3. Run sync

The last step (rsync) is important. We can use sudo iotop to check if rsync is finished or not.

In one instance, I found rsync stalled at some point. When I use md5sum on the source file (exFAT partition), I got an error that the source file has an input/output error.

speed comparison of cp vs rsync

BigData basic: copy & delete folder containing large number of files

Incremental backup: --link-dest

rsync -aP --link-dest=PATHTO/$PREVIOUSBACKUP $SOURCE $CURRENTBACKUP

Ignore existing files

--ignore-existing

Update the remote only if a newer version is on the local filesystem

--update

rsync with exclude files/directories

See http://www.thegeekstuff.com/2011/01/rsync-exclude-files-and-folders/. The key is excluded files are relative to the current directory even we specify the absolute path. For example /path1/path2/file does not mean the file is located under /path1/path2; it means the file is located under ./path1/path2. 

rsync -avz --exclude '/path1/path2/file' source/ destination/

We add add multiple --exclude to exclude more files/directories.

rsync exclude certain file types

rsync -avz --exclude '*.txt' --exclude '~$*' source/ destination/

rsync to exclude hidden files/directories

--exclude=".*"  # exclude both hidden files and directories
--exclude ".*"  # same as above

--exclude ".*/" # exclude hidden directories ONLY

--exclude ".DS_Store"

--exclude ".git" # exclude .git directory ONLY; relative to the directory to be synchronized.

Rsync Copy Hidden Dot Files and Directories Only

Linux / Unix Rsync Copy Hidden Dot Files and Directories Only

rsync with -a option

The -a flag in there stands for “archive,” and it’s important to include. It makes sure that the sync command is recursive (meaning any sub-folders and files inside of old_movies are copied too) and it’s important for preserving all of those modification dates, symbolic links, permissions, and other goodies we talked about earlier.

"-z" option and speed

It will reduce the speed; for example, from 940Mbits/s (117MB/s) to 15MB/s.

delete local files not on remote anymore

Use the --delete option.

What port does rsync use?

https://www.quora.com/What-port-does-rsync-use.

rsync use SSH for connection normally.

Rsync can also operate in a daemon mode, serving and receiving files in the native rsync protocol (using the "rsync://" syntax) and the port number is 873. See https://en.wikipedia.org/wiki/Rsync .

Backup script and SSH Authentication

How To Backup Files From Remote Linux VPS Using Rsync Script

rsync with non-standard port

Use -e option 

rsync -avz -e "ssh -p 23" mydir user@remoteip:

rsync -avz -e "ssh -p 23 -i /home/username/.ssh/idname" mydir user@remoteip:

rsync with progress bar

Use --progress or -P option. The speed is in MB (not Mb). I got about 16MB at home by using Ethernet. 

rsync -avzP file1 file2

The 'rsync' command works on transferring files local to local too.

Or it is better to use -P option which is the same as --partial --progress. When it is used you’ll get a progress dialog at the command line that shows you which file is currently transferring, what percentage of that transfer is complete, and how many more files are left to check. As each file completes, you’ll see an ever-growing list of completed file transfers, which is great for making sure everything transfers successfully. It also allows you to easily resume suspended or interrupted transfers. Combined, you can see how it’ll show you which file was the last one to go, where it failed, and if it failed, give you the option to resume. It’s a pretty powerful combination.

Keep trying to rsync the folder over an unreliable connection

until rsync -avy --delete-after /volume1/backup/ stobokor:.; do sleep 60; done

When rsync fails, a retry will be started after 60s. When rsync completes, the loop will end.

rsync on Windows

Download and install command line rsync from http://www.rsync.net/resources/howto/windows_rsync.html. The website also provides a documentation. Some people are concern about the license issue. The website here provides a link to the free, old but usable version 4.0.5 which is newer than I tested v3.1.0.

Below are my note by using cwrsync v3.1.0 installer got from http://www.rsync.net. 

cd C:\Program Files (x86)\cwRsync\bin
ssh-keygen -t rsa -N ''
rsync -av "/cygdrive/c/Users/brb/.ssh/id_rsa.pub" [email protected]:.ssh/authorized_keys

rsync -av "/cygdrive/c/Users/brb/Downloads/cytokineMC.txt" [email protected]:Downloads/

How to Migrate a Linux Server To New Hardware

How to Migrate a Linux Server To New Hardware

rsync and its GUIs programs

How to Back Up Your Linux System

split large files before transfer/backup

Step 1: back up

This would leave me with the original file (myarchive.tgz) plus a series of files named "myarchive.tgz.00", "myarchive.tgz.01", etc., each no larger than 2G. 

# Method 1
tar -czf myarchive.tgz mydir
split -d -b 2G myarchive.tgz myarchive.tgz.

# Method 2
tar -cz mydir | split -d -b 2G - myarchive.tgz.

md5sum 

md5sum myarchive.tgz.* > myarchive.md5

Step 2: restore 

md5sum -c myarchive.md5

cat myarchive.tgz.* | tar -xz

Top 8 File Backup Apps for Linux

Top 8 File Backup Apps for Linux

  • Bacular
  • UrBackup
  • Clonezilla
  • rsync
  • Déjà Dup
  • Back In Time
  • Timeshift
  • Amanda

13 open source backup solutions

13 open source backup solutions

  • 6 tools: Cronopete, Deja Dup, Rclone, Rdiff-backup, Restic, and Rsync.
  • 13 tools: BorgBackup, UrBackup, LuckyBackup, Casync, Syncthing, Duplicati, Dirvish, Bacula, BackupPC, Amanda, Back in Time, Timeshift, Kup.

LuckyBackup

http://luckybackup.sourceforge.net/

LuckyBackup was featured by MX Linux.

Syncthing (Dropbox alternative), P2P

Duplicati

Rclone

Google

OneDrive

How To Keep OneDrive In Sync With A Folder On Linux Using OneDrive Free Client Fork

rdiff-backup

rdiff-backup is a backup software written in Python that creates reverse incremental backups. See

Dropbox

Three devices for a free account user.

If we don't want to install dropbox software, we can install bash dropbox uploader: http://www.andreafabrizi.it/?dropbox_uploader OR https://github.com/andreafabrizi/Dropbox-Uploader

It allows to upload/download/delete/list files and show info of user. The version I am using is v0.9.7. It works on linux, Windows/Cygwin, Raspberry Pi, etc.

I install it under ~/Downloads/andreafabrizi-Dropbox-Uploader-cdc2466 directory

Instruction with screenshots: http://www.jobnix.in/dropbox-command-line-interface-cli-client/

Sample usages: 

./dropbox_uploader.sh  list /
./dropbox_uploader.sh  upload ~/Desktop/ConfigurateNote.txt

Alternative: Nextcloud

I think I found a Dropbox replacement with Nextcloud... (video)

Alternative: Syncthing

Alternative: Seafile

Seafile on Docker - Self-Hosted Dropbox/Google Drive (video)

Alternative: Duplicati

Alternative: IPFS

Alternative: Cozy

Clone

dd

Clonezilla

back up DVDs

See Create an iso file from a DVD/CD

back up a remote drive using SSH and save the resulting archive to your local machine

# ssh [email protected] "dd if=/dev/sda | gzip -1 -" | dd of=backup.gz

where "-1" (one) indicates the fastest compression method and the dash means ‘standard input’.

Back up a website

Backups vs. Archives

How to Archive Your Data (Virtually) Forever

Wayback machine from archive.org

What Is the Wayback Machine, and Why Is It Important?

Delete files or directories older than XXX days

Suppose we have a cron job to run backup and that creates a lot of directories. We like to erase them to avoid our system running out of space. You can run a space-checking program that'll alert you when the filesystem is low on free space. In addition we can run a cron job that uses find to erase all files. 

cp -p -r ~/Downloads/SomeOldDirectory /tmp/
ls -lt /tmp
# List only
# Older than 2 days. Note the plus sign after "-mtime".
# Quotes in "{}" is to deal with names with spaces
find /tmp -type d -mtime +2 -exec ls -ld "{}" +

find /tmp -type d -mtime +2 -exec rm -rf "{}" +  
# Or if we don't mind some extra warning messages
# find /tmp -type d -mtime +2 -exec rm -rf "{}" \;
ls -lt /tmp

See

Snapshot

Kali Linux adds VM-like snapshot feature to bare-metal installs

Timeshift

Vorta (BorgBackup GUI)

  • BorgBackup 
    sudo apt install borgbackup
borg init --encryption=none backup
borg create --stats backup::snapshot-{utcnow} rstudio/
borg create --stats backup::snapshot-{utcnow} ubuntu-*
mkdir r
borg create --stats backup::snapshot-{utcnow} ubuntu-*
borg prune --keep-last 1 backup/
borg list backup/
borg mount backup r
du -h r

CloudBerry

CloudBerry Backup Protects Files on Windows, Mac, and Linux

Github, Bitbucket, Gitlab

We can use these git services to get real-time data (eg temperature, IP, etc).

Incident Response Plan

What Is an Incident Response Plan?

Retrieved from "https://wiki.taichimd.us/index.php?title=Backup&oldid=37934"