Backup: Difference between revisions
Line 94: | Line 94: | ||
Rsync can also operate in a '''daemon mode''', serving and receiving files in the native rsync protocol (using the "rsync://" syntax) and the port number is 873. See https://en.wikipedia.org/wiki/Rsync . | Rsync can also operate in a '''daemon mode''', serving and receiving files in the native rsync protocol (using the "rsync://" syntax) and the port number is 873. See https://en.wikipedia.org/wiki/Rsync . | ||
== Backup script and SSH Authentication == | |||
[https://ostechnix.com/rsync-script-backup-files-from-remote-linux-vps/ How To Backup Files From Remote Linux VPS Using Rsync Script] | |||
== rsync with non-standard port == | == rsync with non-standard port == |
Revision as of 13:16, 17 October 2023
rsync
Dry run
- --dry-run https://www.tecmint.com/rsync-local-remote-file-synchronization-commands/
- -n https://linux.die.net/man/1/rsync
rsync -avn master.bioconductor.org::release rsync -n --dry-run master.bioconductor.org::release
Install
When using rsync to transfer data remotely, it must be installed on both the source and the destination machine. LibreElec does not include it by default.
I observe that "rsync" and "sshd" on the remote machine were using CPU when I copy from the remote machine to local.
Wildcard/glob
quoting is so important. Bash will replace it with a list of files before rsync is ever called.
# Not working rsync -avz Domain:Dir/myfiles* . # Correct rsync -avz Domain:"Dir/myfiles*" . # OR scp -p Domain:Dir/myfiles* .
Copy large file
If we need to copy large file (say > 4GB), we shall
- format USB drive to NTFS (exFat seems not work)
- Run rsync --progress source dest
- Run sync
The last step (rsync) is important. We can use sudo iotop to check if rsync is finished or not.
In one instance, I found rsync stalled at some point. When I use md5sum on the source file (exFAT partition), I got an error that the source file has an input/output error.
speed comparison of cp vs rsync
BigData basic: copy & delete folder containing large number of files
Incremental backup: --link-dest
rsync -aP --link-dest=PATHTO/$PREVIOUSBACKUP $SOURCE $CURRENTBACKUP
- Time Machine for every Unix out there
- Automating backups on a Raspberry Pi NAS
- How to use advanced rsync for large Linux backups
Ignore existing files
--ignore-existing
Update the remote only if a newer version is on the local filesystem
--update
rsync with exclude files/directories
See http://www.thegeekstuff.com/2011/01/rsync-exclude-files-and-folders/. The key is excluded files are relative to the current directory even we specify the absolute path. For example /path1/path2/file does not mean the file is located under /path1/path2; it means the file is located under ./path1/path2.
rsync -avz --exclude '/path1/path2/file' source/ destination/
We add add multiple --exclude to exclude more files/directories.
rsync exclude certain file types
rsync -avz --exclude '*.txt' --exclude '~$*' source/ destination/
--exclude=".*" # exclude both hidden files and directories --exclude ".*" # same as above --exclude ".*/" # exclude hidden directories ONLY --exclude ".DS_Store" --exclude ".git" # exclude .git directory ONLY; relative to the directory to be synchronized.
Rsync Copy Hidden Dot Files and Directories Only
Linux / Unix Rsync Copy Hidden Dot Files and Directories Only
rsync with -a option
The -a flag in there stands for “archive,” and it’s important to include. It makes sure that the sync command is recursive (meaning any sub-folders and files inside of old_movies are copied too) and it’s important for preserving all of those modification dates, symbolic links, permissions, and other goodies we talked about earlier.
"-z" option and speed
It will reduce the speed; for example, from 940Mbits/s (117MB/s) to 15MB/s.
delete local files not on remote anymore
Use the --delete option.
What port does rsync use?
https://www.quora.com/What-port-does-rsync-use.
rsync use SSH for connection normally.
Rsync can also operate in a daemon mode, serving and receiving files in the native rsync protocol (using the "rsync://" syntax) and the port number is 873. See https://en.wikipedia.org/wiki/Rsync .
Backup script and SSH Authentication
How To Backup Files From Remote Linux VPS Using Rsync Script
rsync with non-standard port
Use -e option
rsync -avz -e "ssh -p 23" mydir user@remoteip: rsync -avz -e "ssh -p 23 -i /home/username/.ssh/idname" mydir user@remoteip:
rsync with progress bar
Use --progress option. The speed is in MB (not Mb). I got about 16MB at home by using Ethernet.
rsync -avz --progress file1 file2
The 'rsync' command works on transferring files local to local too.
Or it is better to use -P option which is the same as --partial --progress. When it is used you’ll get a progress dialog at the command line that shows you which file is currently transferring, what percentage of that transfer is complete, and how many more files are left to check. As each file completes, you’ll see an ever-growing list of completed file transfers, which is great for making sure everything transfers successfully. It also allows you to easily resume suspended or interrupted transfers. Combined, you can see how it’ll show you which file was the last one to go, where it failed, and if it failed, give you the option to resume. It’s a pretty powerful combination.
Keep trying to rsync the folder over an unreliable connection
until rsync -avy --delete-after /volume1/backup/ stobokor:.; do sleep 60; done
When rsync fails, a retry will be started after 60s. When rsync completes, the loop will end.
rsync on Windows
Download and install command line rsync from http://www.rsync.net/resources/howto/windows_rsync.html. The website also provides a documentation. Some people are concern about the license issue. The website here provides a link to the free, old but usable version 4.0.5 which is newer than I tested v3.1.0.
Below are my note by using cwrsync v3.1.0 installer got from http://www.rsync.net.
cd C:\Program Files (x86)\cwRsync\bin ssh-keygen -t rsa -N '' rsync -av "/cygdrive/c/Users/brb/.ssh/id_rsa.pub" [email protected]:.ssh/authorized_keys rsync -av "/cygdrive/c/Users/brb/Downloads/cytokineMC.txt" [email protected]:Downloads/
How to Migrate a Linux Server To New Hardware
How to Migrate a Linux Server To New Hardware
rsync and its GUIs programs
How to Back Up Your Linux System
split large files before transfer/backup
Step 1: back up
This would leave me with the original file (myarchive.tgz) plus a series of files named "myarchive.tgz.00", "myarchive.tgz.01", etc., each no larger than 2G.
# Method 1 tar -czf myarchive.tgz mydir split -d -b 2G myarchive.tgz myarchive.tgz. # Method 2 tar -cz mydir | split -d -b 2G - myarchive.tgz.
md5sum
md5sum myarchive.tgz.* > myarchive.md5
Step 2: restore
md5sum -c myarchive.md5 cat myarchive.tgz.* | tar -xz
Top 8 File Backup Apps for Linux
Top 8 File Backup Apps for Linux
- Bacular
- UrBackup
- Clonezilla
- rsync
- Déjà Dup
- Back In Time
- Timeshift
- Amanda
13 open source backup solutions
13 open source backup solutions
- 6 tools: Cronopete, Deja Dup, Rclone, Rdiff-backup, Restic, and Rsync.
- 13 tools: BorgBackup, UrBackup, LuckyBackup, Casync, Syncthing, Duplicati, Dirvish, Bacula, BackupPC, Amanda, Back in Time, Timeshift, Kup.
LuckyBackup
http://luckybackup.sourceforge.net/
LuckyBackup was featured by MX Linux.
Syncthing (Dropbox alternative), P2P
- https://syncthing.net/. Syncthing is a continuous file synchronization program.
- The Best Dropbox Alternatives: Seafile vs NextCloud vs Syncthing
- Replacing Dropbox with Syncthing
- Docker from linuxserver.io.
- How to Install Syncthing Remote File Synchronization Software on Debian 11
- Syncing your Files Across ALL your Computers via Syncthing (video) from Learn Linux TV 2020/7/10
- 3 machines were set up. However, no "central server".
- One is used as a central location identified by Device ID. Other devices can Add Remote Device by pasting the main device's Device ID and specifying the Shared folders.
- Default folder is ~/Sync
- Change the configuration on ~/.config/syncthing/config.xml so we can use the IP to access the management page
- After the initial setup, the rest operations can be done via a browser
- We can add as many folders as we want. This can be initialized from the central server.
- In Advanced tab, we can choose the Folder Type to be Send & Receive or Send Only or Receive Only
- How to use SyncThing to sync files without the cloud
Duplicati
- https://www.duplicati.com/. Duplicati works with standard protocols like FTP, SSH, WebDAV as well as popular services like Backblaze B2, Tardigrade, Microsoft OneDrive, Amazon S3, Google Drive, box.com, Mega, hubiC and many others.
- Docker from linuxserver.io.
- Duplicati a Set it and Forget it backup tool for local and remote backups of your system!
- 22 awesome open source programs that do everything you need 2023/10
Rclone
- Rclone 1.49 Adds Google Photos Backend, New Web UI 2019-08-29
- From rclone.org
- How to use rclone to automatically back up Google Drive to your local storage
- RCLONE Google Drive Backup
OneDrive
How To Keep OneDrive In Sync With A Folder On Linux Using OneDrive Free Client Fork
rdiff-backup
rdiff-backup is a backup software written in Python that creates reverse incremental backups. See
- https://en.wikipedia.org/wiki/Rdiff-backup
- Turn your old Raspberry Pi into an automatic backup server
- What's new with rdiff-backup version 2?
- rdiff-backup – A Remote Incremental Backup Tool for Linux
Dropbox
Three devices for a free account user.
If we don't want to install dropbox software, we can install bash dropbox uploader: http://www.andreafabrizi.it/?dropbox_uploader OR https://github.com/andreafabrizi/Dropbox-Uploader
It allows to upload/download/delete/list files and show info of user. The version I am using is v0.9.7. It works on linux, Windows/Cygwin, Raspberry Pi, etc.
I install it under ~/Downloads/andreafabrizi-Dropbox-Uploader-cdc2466 directory
Instruction with screenshots: http://www.jobnix.in/dropbox-command-line-interface-cli-client/
Sample usages:
./dropbox_uploader.sh list / ./dropbox_uploader.sh upload ~/Desktop/ConfigurateNote.txt
Alternative: Nextcloud
I think I found a Dropbox replacement with Nextcloud... (video)
Alternative: Syncthing
Alternative: Seafile
Seafile on Docker - Self-Hosted Dropbox/Google Drive (video)
Alternative: Duplicati
Alternative: IPFS
Alternative: Cozy
Clone
- 4 Methods To Clone Your Linux Hard Drive . 1) dd can clone an entire HDD or disk partition, 2) Partimage. partimage does not support the ext4 filesystem, 3) Partclone, and 4) Clonezilla
- Free Drive Cloning Applications. 1) Samsung Data Migration, 2) DiskGenius, and 3) Clonezilla Live.
dd
Clonezilla
- The advanced parameters for restoring mode
- Clonezilla Clone Larger Disk to Smaller Disk vs Easier Way
- CloneZilla – Cloning from a LARGE Windows drive to a SMALLER drive
- Cloning hard disk partition to smaller SSD on laptop
- Running Out of Space? How to Clone Your Linux System Drive to a Larger SSD With CloneZilla
back up DVDs
See Create an iso file from a DVD/CD
back up a remote drive using SSH and save the resulting archive to your local machine
# ssh [email protected] "dd if=/dev/sda | gzip -1 -" | dd of=backup.gz
where "-1" (one) indicates the fastest compression method and the dash means ‘standard input’.
Back up a website
- How to Back Up Your Website Through SSH Command Line
- How to Download an Entire Website for Offline Reading
Backups vs. Archives
How to Archive Your Data (Virtually) Forever
Wayback machine from archive.org
What Is the Wayback Machine, and Why Is It Important?
Delete files or directories older than XXX days
Suppose we have a cron job to run backup and that creates a lot of directories. We like to erase them to avoid our system running out of space. You can run a space-checking program that'll alert you when the filesystem is low on free space. In addition we can run a cron job that uses find to erase all files.
cp -p -r ~/Downloads/SomeOldDirectory /tmp/ ls -lt /tmp # List only # Older than 2 days. Note the plus sign after "-mtime". # Quotes in "{}" is to deal with names with spaces find /tmp -type d -mtime +2 -exec ls -ld "{}" + find /tmp -type d -mtime +2 -exec rm -rf "{}" + # Or if we don't mind some extra warning messages # find /tmp -type d -mtime +2 -exec rm -rf "{}" \; ls -lt /tmp
See
- How to Find or Delete Files Less Than X Minutes Old on Linux
- How to find and delete directory recursively on Linux or Unix-like system
- How to delete all files before a certain date in Linux
- Avoding server disaster
Snapshot
Kali Linux adds VM-like snapshot feature to bare-metal installs
Timeshift
- http://www.teejeetech.in/p/timeshift.html
- How To Backup And Restore Linux With Timeshift
- Linux TimeShift for Backup Tutorial
- How to set up a backup snapshot on Linux Mint (4 min video)
- Taking Linux System Snapshots with Timeshift
Vorta (BorgBackup GUI)
- Securely And Efficiently Backup Data On Linux Or macOS With Vorta (BorgBackup GUI), Vorta BorgBackup GUI Now Available For Install On Linux From Flathub
- BorgBackup
sudo apt install borgbackup borg init --encryption=none backup borg create --stats backup::snapshot-{utcnow} rstudio/ borg create --stats backup::snapshot-{utcnow} ubuntu-* mkdir r borg create --stats backup::snapshot-{utcnow} ubuntu-* borg prune --keep-last 1 backup/ borg list backup/ borg mount backup r du -h r
CloudBerry
CloudBerry Backup Protects Files on Windows, Mac, and Linux
Github, Bitbucket, Gitlab
We can use these git services to get real-time data (eg temperature, IP, etc).