UrBackup

References

• https://www.urbackup.org/, the website.
• UrBackup administration manual.

Tuning

• Performance considerations for larger UrBackup server instances
• Backup server for the raspberry pi
• Internet Mode

Install on Ubuntu

• Install instructions.

Server

sudo add-apt-repository ppa:uroni/urbackup
sudo apt update
sudo apt install urbackup-server

Client

TF=`mktemp` && wget "https://hndl.urbackup.org/Client/2.1.15/UrBackup%20Client%20Linux%202.1.15.sh" -O $TF && sh $TF; rm $TF
# Verifying archive integrity... All good.
# Uncompressing UrBackup Client Installer for Linux  100%  
# Installation of UrBackup Client 2.1.15 to /usr/local ... Proceed ? [Y/n]
# y
# Uncompressing install data...
# Detected Debian (derivative) system
# Detected systemd
# Detected architecture x86_64-linux-glibc
# Installed daemon configuration at /etc/default/urbackupclient...
# Info: Restoring from web interface is disabled per default. Enable by modifying /etc/default/urbackupclient.
# Installing systemd unit...
# Created symlink from /etc/systemd/system/multi-user.target.wants/urbackupclientbackend.service to /lib/systemd/system/urbackupclientbackend.service.
# Starting UrBackup Client service...
# Successfully started client service. Installation complete.
# +Detected Ubuntu LTS. Dattobd supported
# 
# -Detected no btrfs filesystem
# +Detected LVM volumes
# Please select the snapshot mechanism to be used for backups:
# 1) dattobd volume snapshot kernel module from https://github.com/datto/dattobd
# 2) LVM - Logical Volume Manager snapshots
# 4) Use no snapshot mechanism
# 1
# Configured dattobd. Please install dattobd following the instructions at https://github.com/datto/dattobd
# Configured snapshot mechanism via /usr/local/etc/urbackup/snapshot.cfg

# Install dattobd
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 29FF164C
echo "deb https://cpkg.datto.com/repositories $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/datto-linux-agent.list
sudo apt-get update
sudo apt-get install dattobd-dkms dattobd-utils

By default, server connects to client using ports 35621, 35622, 35623. For this to work, we must add the following ufw rules:

--- a/ufw/user.rules
+++ b/ufw/user.rules
@@ -17,6 +17,12 @@
 :ufw-user-limit-accept - [0:0]
 ### RULES ###
 
+### tuple ### allow tcp 35621,35623 0.0.0.0/0 any 0.0.0.0/0 in
+-A ufw-user-input -p tcp -m multiport --dports 35621,35623 -j ACCEPT
+
+### tuple ### allow udp 35622 0.0.0.0/0 any 0.0.0.0/0 in
+-A ufw-user-input -p udp --dport 35622 -j ACCEPT
+
 ### tuple ### allow tcp 135,139,445 0.0.0.0/0 any 0.0.0.0/0 in
 -A ufw-user-input -p tcp -m multiport --dports 135,139,445 -j ACCEPT
 
--- a/ufw/user6.rules
+++ b/ufw/user6.rules
@@ -17,6 +17,12 @@
 :ufw6-user-limit-accept - [0:0]
 ### RULES ###
 
+### tuple ### allow tcp 35621,35623 ::/0 any ::/0 in
+-A ufw6-user-input -p tcp -m multiport --dports 35621,35623 -j ACCEPT
+
+### tuple ### allow udp 35622 ::/0 any ::/0 in
+-A ufw6-user-input -p udp --dport 35622 -j ACCEPT
+
 ### tuple ### allow tcp 135,139,445 ::/0 any ::/0 in
 -A ufw6-user-input -p tcp -m multiport --dports 135,139,445 -j ACCEPT

However, this means that any device can connect and read the files on the client! So, on untrusted network, it is recommended to enable Internet only mode [1]):

echo "internet_server=example.com
internet_server_port=55415
internet_authkey=foobar
internet_mode_enabled=true" > /usr/local/var/urbackup/data/settings.cfg

UrBackup client is not installed as a package. Files are found here:

/etc/default/urbackupclient
/etc/systemd/system/multi-user.target.wants/urbackupclientbackend.service
/lib/systemd/system/urbackupclientbackend.service
/usr/local/bin/urbackupclientctl
/usr/local/etc/urbackup
/usr/local/sbin/uninstall_urbackupclient
/usr/local/sbin/urbackupclientbackend
/usr/local/share/urbackup
/usr/local/var/urbackup
/var/log/urbackupclient.log

Dattobd on client

To enable dattobd snapshot, edit the file /usr/local/etc/urbackup/snapshot.cfg:

#This is a key=value config file for determining the scripts/programs to create snapshots

create_filesystem_snapshot=/usr/local/share/urbackup/dattobd_create_filesystem_snapshot
remove_filesystem_snapshot=/usr/local/share/urbackup/dattobd_remove_filesystem_snapshot

To disable snapshot, remove the file or set the keys to empty value:

#This is a key=value config file for determining the scripts/programs to create snapshots

#create_filesystem_snapshot=/usr/local/share/urbackup/dattobd_create_filesystem_snapshot
#remove_filesystem_snapshot=/usr/local/share/urbackup/dattobd_remove_filesystem_snapshot
create_filesystem_snapshot=
remove_filesystem_snapshot=

To remove snapshot / overlay files left by dattobd /usr/local/share/urbackup/dattobd_remove_filesystem_snapshot:

cat /proc/datto-info
# # cat /proc/datto-info 
# {
# 	"version": "0.9.16",
# 	"devices": [
# 		{
# 			"minor": 0,
# 			"cow_file": "/.datto_3d41c58e-6724-4d47-8981-11c766a08a24_63a1cea102479ac8f40726939d656624778f1a2f7fb6714f",
# 			"block_device": "/dev/dm-1",
# 			"max_cache": 314572800,
# 			"fallocate": 2147483648,
# 			"seq_id": 1,
# 			"uuid": "e8f7f4d4b11549458da6c8cfdad6f93f",
# 			"state": 3
# 		}
# 	]
# }
l /mnt/urbackup_snaps/
# total 4.0K
# drwxr-xr-x 2 root root 48 May  8 05:29 /mnt/urbackup_snaps/63a1cea102479ac8f40726939d656624778f1a2f7fb6714f/
# -rw-r--r-- 1 root root  2 May  8 05:29 /mnt/urbackup_snaps/63a1cea102479ac8f40726939d656624778f1a2f7fb6714f-num
sudo umount /mnt/urbackup_snaps/63a1cea102479ac8f40726939d656624778f1a2f7fb6714f
sudo rmdir /mnt/urbackup_snaps/63a1cea102479ac8f40726939d656624778f1a2f7fb6714f
sudo rm /mnt/urbackup_snaps/63a1cea102479ac8f40726939d656624778f1a2f7fb6714f-num
losetup
# NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
# /dev/loop0         0      0         1  0 /.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_63a1cea102479ac8f40726939d656624778f1a2f7fb6714f
losetup -D
rm /.overlay_2fefd007-3e48-4162-b2c6-45ccdda22f37_63a1cea102479ac8f40726939d656624778f1a2f7fb6714f 
dmsetup ls
# sda5_crypt	(252:0)
# ubuntu--vg-home	(252:3)
# ubuntu--vg-swap_1	(252:2)
# wsnap-63a1cea102479ac8f40726939d656624778f1a2f7fb6714f	(252:4)
# ubuntu--vg-root	(252:1)
dmsetup remove wsnap-63a1cea102479ac8f40726939d656624778f1a2f7fb6714f
dbdctl destroy 0

Configuration

To change location of backup storage:

sudo dpkg-reconfigure urbackup-server

Benchmarks

First backup

• Server: Griffin, backup to NFS /smb/lacie-cloudbox/beq06659/urbackup.
• Client: zavcxl0005, backup of /home/data /home/peetersm

• 129247M (135525883904), 80260 directories, 697949 files.
• Using dattobd.

• Default urbackup configuration.
• Log:

• Activity: Required time 185 min, Used storage 133.56 GB.
• Log: 142.388 GB, average speed: 110.523MBit/s
• Log: Time taken for backing up client zavcxl0005: 5h 27m 11s
• BACKUP INTERRUPTED because backup storage disk FULL!

Note:

• After reboot of Griffin, UrBackup took ~10min to rebuild index and delete the last backup! However the files are still there. Hopefully next backup would be faster...
• Stats on lacie-cloudbox:

• Collecting these stats is very slow.
• There are 3.5x more files in the repository.
• Storage is not compressed

du -hs /shares/beq06659/urbackup
# 144G    .                     # Took >10min to get
find -type f|wc -l
# 2426985                       # Took >10min to get! This is 3.5x the number of files to backup!!!
find -type d|wc -l
# 260410                        # Took >10min to get! This is 3.25x the number of directories to backup!!!

• On the client, log file /var/log/urbackupclient.log full of errors like

2017-05-08 05:41:18: ERROR: Error stating file "/mnt/urbackup_snaps/63a1cea102479ac8f40726939d656624778f1a2f7fb6714f//usr/share/help/es/gnome-help/prefs-display.page" to get file tokens. Errno: 2
2017-05-08 05:41:18: ERROR: Error stating file "/mnt/urbackup_snaps/63a1cea102479ac8f40726939d656624778f1a2f7fb6714f//usr/share/help/es/gnome-help/prefs-language.page" to get file tokens. Errno: 2
2017-05-08 05:41:18: ERROR: Error stating file "/mnt/urbackup_snaps/63a1cea102479ac8f40726939d656624778f1a2f7fb6714f//usr/share/help/es/gnome-help/prefs-sharing.page" to get file tokens. Errno: 2

Todo:

• Disable following symlinks.
• Backup with and without dattobd.

Second backup

• Server: Griffin, backup to NFS /smb/lacie-cloudbox/beq06659/urbackup.
• Client: zavcxl0005, backup of /home/data /home/peetersm

• 104924M (110020796416), 78008 directories, 674926 files.
• Without snapshot.

• On the client:

urbackupclientctl add-backupdir -f -d /home/data
urbackupclientctl add-backupdir -f -d /home/peetersm
urbackupclientctl list-backupdirs
urbackupclientctl start --full
# Waiting for server to start backup... done
# Preparing... done
# [=====================================] 100% 101.6 GB/101.676 GB at 129.082 MBit/s    
# Completed successfully.

• On the client, log file /var/log/urbackupclient.log still some error like the following, but much much less

2017-05-09 00:25:51: ERROR: Error stating file "/home/data/work/_git/_ext/the_silver_searcher/compile" to get file tokens. Errno: 2
2017-05-09 00:25:51: ERROR: Error stating file "/home/data/work/_git/_ext/the_silver_searcher/config.guess" to get file tokens. Errno: 2
2017-05-09 00:25:51: ERROR: Error stating file "/home/data/work/_git/_ext/the_silver_searcher/config.sub" to get file tokens. Errno: 2

These are due to broken symlinks, so nothing to worry about.

Statistics

• First full backup: Required 107min, Used 97.18 GB, Transferred 101.725 GB, average speed: 136.985 MBit/s, Total 177m.
• Second incremental backup: Required 5min, Used 144.75 MB, Transferred 182.907 MB, average speed: 5.57627 MBit/s, Total 5m.

du -hs
# 104G  .
time find -type d|wc -l               # real  5m39.850s
# 174629
time find -type f|wc -l               # real  5m47.236s
# 1497675
du -sBM ~urbackup
# 1339M /var/urbackup

• Third incremental backup: Required 17 min, Used 13.21 MB, Transferred 13.2972 MB, average speed: 115.899 MBit/s, Total 18m.
• Statistics: Total storage 110.54GB.

time du -hs                           # real 14m16.070s
# 117G  .
du -sBM ~urbackup
# 1370M /var/urbackup

• Fourth incremental backup: Required 6 min, Used 238.8 MB, Transferred 277.002 MB, average speed: 7.66078 MBit/s, Total 6m.

• Fifth incremental backup: Required 17 min, Used 13.43 GB, Transferred 13.4788 GB, average speed: 115.231 MBit/s, Total 18m.

• Sixth incremental backup: Required 4 min, Used 182.44 MB, Transferred 218.76 MB, average speed: 10.1838 MBit/s, Total 4m.
• Statistics: Total storage 124.39GB.

du -sB1                               # real 18m18.728s
# 140828954624  .
# 131G  .
time find -type d|wc -l               # real    7m36.431s
# 244426
time find -type f|wc -l               # real    7m55.833s
# 1933327
du -sBM ~urbackup
# 1422M /var/urbackup

• Seventh incremental backup: Required 17 min, Used 13.43 GB, Transferred 13.8994 GB, average speed: 120.863 MBit/s, Total 18m.

• Enable snapshot with dattobd.
• Eighth incremental backup: Required 7 min, Used 152.89 MB, Transferred 193.38 MB, average speed: 4.74 MBit/s, Total 8m.

du -sB1                               # real 19m1.994s
# 155743612928  .
# 145G  .
time find -type d|wc -l               # real    8m19.903s
# 281684
time find -type f|wc -l               # real    8m44.226s
# 2157340
du -sBM ~urbackup
# ----M /var/urbackup

• Ninth incremental backup: Required 17 min, Used 13.29 GB, Transferred 13.3237 GB, average speed: 118.711 MBit/s, Total 18m.

du -sB1                               # real 
# xxxxxxxxxxxx  .
# xxxG  .
time find -type d|wc -l               # real    
# xxxxxx
time find -type f|wc -l               # real    
# xxxxxxx
du -sBM ~urbackup
# 1602M /var/urbackup