Bup

References

• Bup on GitHub
• Bup website

Bup reviews:

• Bup - towards the perfect backup.

Also details how to setup snapshot on Windows using vscsc to make consistent backups.

• Git-based backup with bup

Someone commented that par2 should be replaced by zfec.

Install on Lacie-Cloudbox

git clone https://github.com/bup/bup
ipkg install attr libacl
ipkg install python27 py27-tornado py27-pip
which python
which python2
# optware python only available as "python2", and pip as "pip2.7"
pip2.7 install fuse-python
# pip2.7 install xattr libacl
# # error: /opt/local/lib/python2.5/site-packages (in --site-dirs) is not on sys.path
# export PYTHONPATH=/opt/local/lib/python2.5/site-packages
# # see https://forum.qnap.com/viewtopic.php?t=73608

pip2.7 install pyxattr pylibacl       # Do NOT install xattr

export PYTHON=/opt/bin/python2        # To avoid using /usr/bin/python, see bup doc.
make
make test                             # Many issues... see below.
make install PREFIX=/opt

Houston, we have a problem. We can reproduce as follows:

cat bup_intl_issue.py
# import posix1e;           # From package pylibacl
python bup_intl_issue.py
# /opt/bin/python2.7: symbol 'libintl_gettext': can't resolve symbol
# Traceback (most recent call last):
#   File "bup_intl_issue.py", line 1, in <module>
#     import posix1e;
# ImportError: unknown dlopen() error

We can fix the issue using the LD_PRELOAD trick though:

LD_PRELOAD=/opt/lib/libintl.so python2.7 bup_intl_issue.py

But this trick does not work when calling bup.

Let's postpone that. I've another problem when installing fuse-python:

pip2.7 install fuse-python
# Collecting fuse-python
#   Using cached fuse-python-0.2.tar.gz
#     Complete output from command python setup.py egg_info:
#     pkg-config unavailable, build terminated
#
#     ----------------------------------------
# Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-sKLWYV/fuse-python/

To run bup via ssh, we must add it to the PATH.

• Make sure that PermitUserEnvironment yes is in the file sshd_config on the server.
• Add to ~/.ssh/environment:

PATH=/opt/bin:/opt/local/bin:/opt/sbin:$PATH

Install on Raspberry-pi

git clone https://github.com/bup/bup
sudo apt-get install python2.7-dev python-fuse python-pyxattr python-pylibacl linux-libc-dev acl attr python-tornado pandoc
make
make test
sudo make install           # First run make install before checkinstall
sudo checkinstall           # ... or checkinstall fails

Server config

.ssh/authorized_keys

Bup connects to remote backup server using git. When connecting, the client assumes that the server runs a special bup shell, bup server. For this we must configure the file .ssh/authorized_keys on the server as follows:

command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa ...

Usage

Now, from the client:

bup init -r server:/path/to/bup/dir
# Initialized empty Git repository in /home/localuser/.bup/
#
# Initialized empty Git repository in /path/to/bup/dir

Benchmarks

TBC

du -hs /var/www/datasheet/
# 321M	/var/www/datasheet/
#

bup init -r laciebup:/shares/beq06659/bup
bup index /var/www/datasheet
time bup save -r laciebup:/shares/beq06659/bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 481300/481300, done.
# 
# real	0m36.811s
# user	0m5.880s
# sys	0m0.652s
# 
bup init -r griffinbup:/smb/lacie-cloudbox/beq06659/bup
bup index /var/www/datasheet
time bup save -r griffinbup:/smb/lacie-cloudbox/beq06659/bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 481300/481300, done.
# 
# real	0m9.190s
# user	0m6.660s
# sys	0m0.684s
# 
bup init -r griffinbup:.bup
bup index /var/www/datasheet
time bup save -r griffinbup:.bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Receiving index from server: 481300/481300, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 1100/1100, done.    
# 
# real	0m4.311s
# user	0m2.756s
# sys	0m0.392s
bup init -r pibup:.bup
bup index /var/www/datasheet
time bup save -r pibup:.bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 481300/481300, done.         
# 
# real	1m6.180s
# user	0m6.904s
# sys	0m0.604s
# 
bup init -r pibup:/smb/lacie-cloudbox/beq06659/bup
bup index /var/www/datasheet
time bup save -r pibup:/smb/lacie-cloudbox/beq06659/bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 481300/481300, done.         
# 
# real	1m1.208s
# user	0m6.624s
# sys	0m0.564s

Troubleshoot

fsync(...) - Invalid argument

time bup save -r griffinbup:/smb/lacie-cloudbox/beq06659/bup -n var-www-datasheet /var/www/datasheet
# Traceback (most recent call last):               
#   ...
#   File "/usr/local/lib/bup/bup/git.py", line 776, in _end
#     os.fsync(self.parentfd)
# OSError: [Errno 22] Invalid argument
#   ...
# bup.client.ClientError: server exited unexpectedly with code 1

fsync() is not supported on Samba fs, for directory fd. See [1], [2].

The fix from borgbackup:

+import errno
# ...
         os.fsync(self.parentfd)
+    except OSError as os_error:
+        # Some network filesystems don't support this and fail with EINVAL.
+        # Other error codes (e.g. EIO) shouldn't be silenced.
+        if os_error.errno != errno.EINVAL:
+            raise

'init-dir: command not found' on 'bup init'

bup init -r griffin:/smb/lacie-cloudbox/beq06659/bup
# Initialized empty Git repository in /home/peetersm/.bup/
# 
# -bash: line 1: init-dir: command not found
# ...

The server must be configured to start a bup server shell when connecting with ssh. This can be achieved by configuring the file ~/.ssh/authorized_keys as follows:

command="/path/to/bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa ...

On the mailing list, people uses a more complex line:

from="198.51.100.123,2001:db8::123",command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa ...