Bup: Difference between revisions

From miki
Jump to navigation Jump to search
 
(11 intermediate revisions by the same user not shown)
Line 18: Line 18:
which python2
which python2
# optware python only available as "python2", and pip as "pip2.7"
# optware python only available as "python2", and pip as "pip2.7"
pip2.7 install fuse
pip2.7 install fuse-python
pip2.7 install xattr libacl
# pip2.7 install xattr libacl
# error: /opt/local/lib/python2.5/site-packages (in --site-dirs) is not on sys.path
# # error: /opt/local/lib/python2.5/site-packages (in --site-dirs) is not on sys.path
export PYTHONPATH=/opt/local/lib/python2.5/site-packages
# export PYTHONPATH=/opt/local/lib/python2.5/site-packages
# see https://forum.qnap.com/viewtopic.php?t=73608
# # see https://forum.qnap.com/viewtopic.php?t=73608


pip2.7 install pyxattr pylibacl # Do NOT install xattr
pip2.7 install pyxattr pylibacl # Do NOT install xattr


export PYTHON=/opt/bin/python2 # To avoid using /usr/bin/python, see bup doc.
export PYTHON=/opt/bin/python2 # To avoid using /usr/bin/python, see bup doc.
make
make test # Many issues... see below.
make install PREFIX=/opt
</source>
Houston, we have a problem. We can reproduce as follows:
<source lang=bash>
cat bup_intl_issue.py
# import posix1e; # From package pylibacl
python bup_intl_issue.py
# /opt/bin/python2.7: symbol 'libintl_gettext': can't resolve symbol
# Traceback (most recent call last):
# File "bup_intl_issue.py", line 1, in <module>
# import posix1e;
# ImportError: unknown dlopen() error
</source>
We can fix the issue using the <code>LD_PRELOAD</code> trick though:
<source lang="bash">
LD_PRELOAD=/opt/lib/libintl.so python2.7 bup_intl_issue.py
</source>
But this trick does not work when calling <code>bup</code>.

Let's postpone that. I've another problem when installing <code>fuse-python</code>:
<source lang=bash>
pip2.7 install fuse-python
# Collecting fuse-python
# Using cached fuse-python-0.2.tar.gz
# Complete output from command python setup.py egg_info:
# pkg-config unavailable, build terminated
#
# ----------------------------------------
# Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-sKLWYV/fuse-python/
</source>

To run <code>bup</code> via ssh, we must add it to the PATH.
* Make sure that <code>PermitUserEnvironment yes</code> is in the file {{file|sshd_config}} on the server.
* Add to {{file|~/.ssh/environment}}:
<source lang="bash">
PATH=/opt/bin:/opt/local/bin:/opt/sbin:$PATH
</source>

== Install on Raspberry-pi ==
<source lang="bash">
git clone https://github.com/bup/bup
sudo apt-get install python2.7-dev python-fuse python-pyxattr python-pylibacl linux-libc-dev acl attr python-tornado pandoc
make
make
make test
make test
sudo make install # First run make install before checkinstall
sudo checkinstall # ... or checkinstall fails
</source>

== Server config ==

;.ssh/authorized_keys
Bup connects to remote backup server using git. When connecting, the client assumes that the server runs a special bup shell, <code>bup server</code>. For this we must configure the file {{file|.ssh/authorized_keys}} on the server as follows:

<source lang="bash">
command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa ...
</source>

== Usage ==

Now, from the client:
<source lang="bash">
bup init -r server:/path/to/bup/dir
# Initialized empty Git repository in /home/localuser/.bup/
#
# Initialized empty Git repository in /path/to/bup/dir
</source>

== Benchmarks ==
TBC

<source lang="bash">
du -hs /var/www/datasheet/
# 321M /var/www/datasheet/
# </source>

<source lang="text">
bup init -r laciebup:/shares/beq06659/bup
bup index /var/www/datasheet
time bup save -r laciebup:/shares/beq06659/bup -n var-www-datasheet /var/www/datasheet
#
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.
# Receiving index from server: 481300/481300, done.
#
# real 0m36.811s
# user 0m5.880s
# sys 0m0.652s
#
bup init -r griffinbup:/smb/lacie-cloudbox/beq06659/bup
bup index /var/www/datasheet
time bup save -r griffinbup:/smb/lacie-cloudbox/beq06659/bup -n var-www-datasheet /var/www/datasheet
#
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.
# Receiving index from server: 481300/481300, done.
#
# real 0m9.190s
# user 0m6.660s
# sys 0m0.684s
#
bup init -r griffinbup:.bup
bup index /var/www/datasheet
time bup save -r griffinbup:.bup -n var-www-datasheet /var/www/datasheet
#
# Reading index: 115, done.
# Receiving index from server: 481300/481300, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.
# Receiving index from server: 1100/1100, done.
#
# real 0m4.311s
# user 0m2.756s
# sys 0m0.392s
bup init -r pibup:.bup
bup index /var/www/datasheet
time bup save -r pibup:.bup -n var-www-datasheet /var/www/datasheet
#
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.
# Receiving index from server: 481300/481300, done.
#
# real 1m6.180s
# user 0m6.904s
# sys 0m0.604s
#
bup init -r pibup:/smb/lacie-cloudbox/beq06659/bup
bup index /var/www/datasheet
time bup save -r pibup:/smb/lacie-cloudbox/beq06659/bup -n var-www-datasheet /var/www/datasheet
#
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.
# Receiving index from server: 481300/481300, done.
#
# real 1m1.208s
# user 0m6.624s
# sys 0m0.564s
</source>

== Troubleshoot ==
=== fsync(...) - Invalid argument ===
<source lang="bash">
time bup save -r griffinbup:/smb/lacie-cloudbox/beq06659/bup -n var-www-datasheet /var/www/datasheet
# Traceback (most recent call last):
# ...
# File "/usr/local/lib/bup/bup/git.py", line 776, in _end
# os.fsync(self.parentfd)
# OSError: [Errno 22] Invalid argument
# ...
# bup.client.ClientError: server exited unexpectedly with code 1
</source>

<code>fsync()</code> is not supported on Samba fs, for directory fd. See [https://groups.google.com/forum/#!searchin/bup-list/samba%7Csort:relevance/bup-list/5rqPrlo_6h8/XYENSGjoAwAJ], [https://github.com/borgbackup/borg/issues/1287].

The fix from [https://github.com/borgbackup/borg/pull/1288/commits/0005023a7354856fafcea144110bcc2b42accd40 borgbackup]:
<source lang="python">
+import errno
# ...
os.fsync(self.parentfd)
+ except OSError as os_error:
+ # Some network filesystems don't support this and fail with EINVAL.
+ # Other error codes (e.g. EIO) shouldn't be silenced.
+ if os_error.errno != errno.EINVAL:
+ raise
</source>

=== 'init-dir: command not found' on 'bup init'===

<source lang="bash">
bup init -r griffin:/smb/lacie-cloudbox/beq06659/bup
# Initialized empty Git repository in /home/peetersm/.bup/
#
# -bash: line 1: init-dir: command not found
# ...
</source>

The server must be configured to start a <code>bup server</code> shell when connecting with ssh.
This can be achieved by configuring the file {{file|~/.ssh/authorized_keys}} as follows:

<source lang="bash">
command="/path/to/bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa ...
</source>

On the mailing list, people uses a more complex line:
<source lang="bash">
from="198.51.100.123,2001:db8::123",command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa ...
</source>
</source>

Latest revision as of 22:56, 1 May 2017

References

Bup reviews:

Also details how to setup snapshot on Windows using vscsc to make consistent backups.
Someone commented that par2 should be replaced by zfec.

Install on Lacie-Cloudbox

git clone https://github.com/bup/bup
ipkg install attr libacl
ipkg install python27 py27-tornado py27-pip
which python
which python2
# optware python only available as "python2", and pip as "pip2.7"
pip2.7 install fuse-python
# pip2.7 install xattr libacl
# # error: /opt/local/lib/python2.5/site-packages (in --site-dirs) is not on sys.path
# export PYTHONPATH=/opt/local/lib/python2.5/site-packages
# # see https://forum.qnap.com/viewtopic.php?t=73608

pip2.7 install pyxattr pylibacl       # Do NOT install xattr

export PYTHON=/opt/bin/python2        # To avoid using /usr/bin/python, see bup doc.
make
make test                             # Many issues... see below.
make install PREFIX=/opt

Houston, we have a problem. We can reproduce as follows:

cat bup_intl_issue.py
# import posix1e;           # From package pylibacl
python bup_intl_issue.py
# /opt/bin/python2.7: symbol 'libintl_gettext': can't resolve symbol
# Traceback (most recent call last):
#   File "bup_intl_issue.py", line 1, in <module>
#     import posix1e;
# ImportError: unknown dlopen() error

We can fix the issue using the LD_PRELOAD trick though:

LD_PRELOAD=/opt/lib/libintl.so python2.7 bup_intl_issue.py

But this trick does not work when calling bup.

Let's postpone that. I've another problem when installing fuse-python:

pip2.7 install fuse-python
# Collecting fuse-python
#   Using cached fuse-python-0.2.tar.gz
#     Complete output from command python setup.py egg_info:
#     pkg-config unavailable, build terminated
#
#     ----------------------------------------
# Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-sKLWYV/fuse-python/

To run bup via ssh, we must add it to the PATH.

  • Make sure that PermitUserEnvironment yes is in the file sshd_config on the server.
  • Add to ~/.ssh/environment:
PATH=/opt/bin:/opt/local/bin:/opt/sbin:$PATH

Install on Raspberry-pi

git clone https://github.com/bup/bup
sudo apt-get install python2.7-dev python-fuse python-pyxattr python-pylibacl linux-libc-dev acl attr python-tornado pandoc
make
make test
sudo make install           # First run make install before checkinstall
sudo checkinstall           # ... or checkinstall fails

Server config

.ssh/authorized_keys

Bup connects to remote backup server using git. When connecting, the client assumes that the server runs a special bup shell, bup server. For this we must configure the file .ssh/authorized_keys on the server as follows:

command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa ...

Usage

Now, from the client:

bup init -r server:/path/to/bup/dir
# Initialized empty Git repository in /home/localuser/.bup/
#
# Initialized empty Git repository in /path/to/bup/dir

Benchmarks

TBC

du -hs /var/www/datasheet/
# 321M	/var/www/datasheet/
#
bup init -r laciebup:/shares/beq06659/bup
bup index /var/www/datasheet
time bup save -r laciebup:/shares/beq06659/bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 481300/481300, done.
# 
# real	0m36.811s
# user	0m5.880s
# sys	0m0.652s
# 
bup init -r griffinbup:/smb/lacie-cloudbox/beq06659/bup
bup index /var/www/datasheet
time bup save -r griffinbup:/smb/lacie-cloudbox/beq06659/bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 481300/481300, done.
# 
# real	0m9.190s
# user	0m6.660s
# sys	0m0.684s
# 
bup init -r griffinbup:.bup
bup index /var/www/datasheet
time bup save -r griffinbup:.bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Receiving index from server: 481300/481300, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 1100/1100, done.    
# 
# real	0m4.311s
# user	0m2.756s
# sys	0m0.392s
bup init -r pibup:.bup
bup index /var/www/datasheet
time bup save -r pibup:.bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 481300/481300, done.         
# 
# real	1m6.180s
# user	0m6.904s
# sys	0m0.604s
# 
bup init -r pibup:/smb/lacie-cloudbox/beq06659/bup
bup index /var/www/datasheet
time bup save -r pibup:/smb/lacie-cloudbox/beq06659/bup -n var-www-datasheet /var/www/datasheet
# 
# Reading index: 115, done.
# Saving: 100.00% (327721/327721k, 115/115 files), done.    
# Receiving index from server: 481300/481300, done.         
# 
# real	1m1.208s
# user	0m6.624s
# sys	0m0.564s

Troubleshoot

fsync(...) - Invalid argument

time bup save -r griffinbup:/smb/lacie-cloudbox/beq06659/bup -n var-www-datasheet /var/www/datasheet
# Traceback (most recent call last):               
#   ...
#   File "/usr/local/lib/bup/bup/git.py", line 776, in _end
#     os.fsync(self.parentfd)
# OSError: [Errno 22] Invalid argument
#   ...
# bup.client.ClientError: server exited unexpectedly with code 1

fsync() is not supported on Samba fs, for directory fd. See [1], [2].

The fix from borgbackup:

+import errno
# ...
         os.fsync(self.parentfd)
+    except OSError as os_error:
+        # Some network filesystems don't support this and fail with EINVAL.
+        # Other error codes (e.g. EIO) shouldn't be silenced.
+        if os_error.errno != errno.EINVAL:
+            raise

'init-dir: command not found' on 'bup init'

bup init -r griffin:/smb/lacie-cloudbox/beq06659/bup
# Initialized empty Git repository in /home/peetersm/.bup/
# 
# -bash: line 1: init-dir: command not found
# ...

The server must be configured to start a bup server shell when connecting with ssh. This can be achieved by configuring the file ~/.ssh/authorized_keys as follows:

command="/path/to/bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa ...

On the mailing list, people uses a more complex line:

from="198.51.100.123,2001:db8::123",command="BUP_DEBUG=0 BUP_FORCE_TTY=3 bup server",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty ssh-rsa ...