Burp: Difference between revisions

From miki
Jump to navigation Jump to search
(Created page with "'''[http://burp.grke.org/index.html Burp]''' is a network backup and restore program. It attempts to reduce network traffic and the amount of space that is used by each backup...")
 
 
(6 intermediate revisions by the same user not shown)
Line 6: Line 6:


== Build on Lacie-CloudBox ==
== Build on Lacie-CloudBox ==

=== Build on Optware ===
Links:
Links:
* [https://gist.github.com/mmv-ru/98c435f591355741f0c4 Burp on GNAP]
* [https://gist.github.com/mmv-ru/98c435f591355741f0c4 Burp on GNAP]
Line 77: Line 79:
<source lang="bash">
<source lang="bash">
make
make
# ... error about missing aclocal...
</source>
Digging further, we see that <code>aclocal</code> does not work because it requires Perl built with threads enabled. For this we need Perl 5.22 from Optware-ng

=== Build on Optware-ng ===
To install
<source lang="bash">
# Install dependencies
ipkg install openssl-dev librsync zlib
ipkg install gcc
ipkg install make automake autoconf libtool binutils autoconf
ipkg install optware-devel

# Install uthash headers in /opt/include:
git clone https://github.com/troydhanson/uthash.git
cp uthash/src/ut* /opt/include/

# Clone git repository
git clone https://github.com/grke/burp.git
cd burp
git checkout 2.1.6

# Generate the autoconf files
autoreconf -vif

# Configure
./configure --prefix=/opt --sysconfdir=/opt/etc/burp --localstatedir=/opt/var

# Make
make

# Install
make install
# Copied/created the following files/directories:
# /opt/bin/vss_strip
# /opt/sbin/burp
# /opt/sbin/burp_ca
# /opt/share/burp
# /opt/share/doc/burp
# /opt/share/man/man8/{bedup.8,bsigs.8,bsparse.8,burp.8,burp_ca.8,vss_strip.8}

# Install config
make install-configs
# Copied/created the following files/directories:
# /opt/etc/burp

# If we want to create a .deb package:
# (Note: if fail, run make install install-configs w/o checkinstall first)
sudo checkinstall make install install-configs
</source>

Edit the following files:
* {{file|/etc/burp/burp-server.conf}} (for backup server)
* {{file|/etc/burp/burp.conf}} (for backup client)

For each client ''cname'' on the server (see <code>cname</code> in {{file|burp.conf}}), edit the file:
* {{file|/etc/burp/clientconfdir/cname}}.

Start the server with:
<source lang="bash">
sudo burp -c /etc/burp/burp-server.conf
</source>

On the client, start a backup with:
<source lang="bash">
sudo burp -a b
</source>

Monitor ongoing backups with:
<source lang="bash">
sudo burp -s
</source>

== Benchmarks ==
<source lang="bash">
### On the server
egrep "(protocol|ratelimit|comp|ssl_ciphers).*=" /opt/etc/burp/burp-server.conf
pkill burp && rm -rf /opt/var/spool/burp/* && burp -c /opt/etc/burp/burp-server.conf

### On the client
egrep "(protocol|ratelimit|comp|ssl_ciphers).*=" /etc/burp/burp.conf
sudo burp -a b
</source>

;/var/www/datasheet on zavcxl0005 -- no compression, protocol 2
<source lang="bash">
du --apparent-size -BM -s /var/www/datasheet/
# 92M /var/www/datasheet/
du -BM -s /var/www/datasheet/
# 92M /var/www/datasheet/
</source>

<source lang="bash">
### On the server
# protocol = 2
# compression = zlib0
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_compression = zlib0
sudo burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 22:35:07
# End time: 2017-04-26 22:36:24
# Time taken: 01:17
# New Changed Duplicate Deleted Total | Scanned
# ------------------------------------------------------------
# Files: 0 0 0 0 0 | 11
# Directories: 0 0 0 0 0 | 6
# Blocks: 14005 0 28811 0 42816 | 0
# Grand total: 14005 0 28811 0 42816 | 17
# ------------------------------------------------------------
#
# Messages: 0
# Warnings: 0
#
# Bytes estimated: 95598630 (91.17 MB)
# Bytes in backup: 0
# Bytes received: 109011 (106.46 KB)
# Bytes sent: 96078512 (91.63 MB)
# --------------------------------------------------------------------------------
# 2017-04-26 22:36:24: burp[18005] End backup
# 2017-04-26 22:36:24: burp[18005] backup finished ok
</source>
* CPU usage on server: '''95% average'''.
* transfer time: '''77s'''.
* transfer speed: '''1.2MB/s'''.

;/var/www/datasheet on zavcxl0005 - no compression, no encryption, protocol 2
<source lang="bash">
### On the server
# protocol = 2
# compression = zlib0
# compression = zlib0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
sudo burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 22:52:53
# End time: 2017-04-26 22:53:58
# Time taken: 01:05
# New Changed Duplicate Deleted Total | Scanned
# ------------------------------------------------------------
# Files: 0 0 0 0 0 | 11
# Directories: 0 0 0 0 0 | 6
# Blocks: 14005 0 28023 0 42028 | 0
# Grand total: 14005 0 28023 0 42028 | 17
# ------------------------------------------------------------
#
# Messages: 0
# Warnings: 0
#
# Bytes estimated: 95598630 (91.17 MB)
# Bytes in backup: 0
# Bytes received: 109011 (106.46 KB)
# Bytes sent: 96078512 (91.63 MB)
# --------------------------------------------------------------------------------
# 2017-04-26 22:53:58: burp[22408] End backup
# 2017-04-26 22:53:58: burp[22408] backup finished ok
</source>
* CPU usage on server: '''95% average'''.
* transfer time: '''65s'''.
* transfer speed: '''1.4MB/s'''.

<source lang="bash">
### On the server
# protocol = 1
# compression = zlib0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 23:12:07
# End time: 2017-04-26 23:12:23
# Time taken: 00:16
# New Changed Duplicate Deleted Total | Scanned
# ------------------------------------------------------------
# Files: 11 0 0 0 11 | 11
# Directories: 0 0 0 0 0 | 6
# Grand total: 11 0 0 0 11 | 17
# ------------------------------------------------------------
#
# Messages: 0
# Warnings: 0
#
# Bytes estimated: 95598630 (91.17 MB)
# Bytes in backup: 95598630 (91.17 MB)
# Bytes received: 1844 (1.80 KB)
# Bytes sent: 95720309 (91.29 MB)
# --------------------------------------------------------------------------------
# 2017-04-26 23:12:23: burp[23561] Phase 2 end (send file data)
# 2017-04-26 23:12:23: burp[23561] backup finished ok
</source>
* CPU usage on server: '''95% average'''.
* transfer time: '''16s'''.
* transfer speed: '''5.7MB/s'''.

; Increasing ratelimit
* No effect on any protocol.

; Protocol 2 - Backup again after duplicating all files
<source lang="bash">
### On the server
# protocol = 2
# compression = zlib0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
sudo cp -r /var/www/datasheet/arm /var/www/datasheet/arm2 # We make a redundant copy inside
sudo burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 23:22:22
# End time: 2017-04-26 23:22:35
# Time taken: 00:13
# New Changed Duplicate Deleted Total | Scanned
# ------------------------------------------------------------
# Files: 0 0 0 0 0 | 22
# Directories: 0 0 0 0 0 | 11
# Blocks: 0 0 18024 0 18024 | 0
# Grand total: 0 0 18024 0 18024 | 33
# ------------------------------------------------------------
#
# Messages: 0
# Warnings: 0
#
# Bytes estimated: 191197260 (182.34 MB)
# Bytes in backup: 0
# Bytes received: 1156 (1.13 KB)
# Bytes sent: 412007 (402.35 KB)
# --------------------------------------------------------------------------------
# 2017-04-26 23:22:35: burp[24000] End backup
# 2017-04-26 23:22:35: burp[24000] backup finished ok
</source>
* CPU usage on server: '''95% average'''.
* transfer time: '''13s'''.
* transfer speed: '''14.0MB/s'''.

; Protocol 1 - Backup again after duplicating all files
<source lang="bash">
### On the server
# protocol = 1
# compression = zlib0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
sudo cp -r /var/www/datasheet/arm /var/www/datasheet/arm2 # We make a redundant copy inside
sudo burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 23:28:13
# End time: 2017-04-26 23:28:30
# Time taken: 00:17
# New Changed Duplicate Deleted Total | Scanned
# ------------------------------------------------------------
# Files: 11 0 0 0 11 | 22
# Directories: 0 0 0 0 0 | 11
# Grand total: 11 0 0 0 11 | 33
# ------------------------------------------------------------
#
# Messages: 0
# Warnings: 0
#
# Bytes estimated: 191197260 (182.34 MB)
# Bytes in backup: 95598630 (91.17 MB)
# Bytes received: 1844 (1.80 KB)
# Bytes sent: 95722406 (91.29 MB)
# --------------------------------------------------------------------------------
# 2017-04-26 23:28:30: burp[24200] Phase 2 end (send file data)
# 2017-04-26 23:28:30: burp[24200] backup finished ok
#
</source>
* CPU usage on server: '''95% average'''.
* transfer time: '''17s'''.
* transfer speed: '''10.7MB/s'''.

== Troubleshoot ==
=== SSL connect error ===
<source lang="bash">
sudo burp -a b
# 2017-04-25 23:09:47: burp[30776] SSL connect error
# 140585554515608:error:1409441B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error:s3_pkt.c:1487:SSL alert number 51
</source>
This happens when using old certificates on backup client, typically after changing the configuration to point to another backup server [https://sourceforge.net/p/burp/mailman/message/34312368/].
<source lang="bash">
cd /etc/burp
rm ssl_cert_ca.pem ssl_cert-client.key ssl_cert-client.pem
</source>
</source>

Latest revision as of 21:32, 26 April 2017

Burp is a network backup and restore program. It attempts to reduce network traffic and the amount of space that is used by each backup.

References

Build on Lacie-CloudBox

Build on Optware

Links:

Example of install log on QNAP

Install:

  • Clone git
# Clone git repository
git clone https://github.com/grke/burp.git
cd burp
git checkout 2.1.6

# Generate the autoconf files
autoreconf -vif
# ...
# Compilation failed in require at /opt/share/automake-1.12/Automake/ChannelDefs.pm line 23.
# ...
# autoreconf: aclocal failed with exit status: 255
  • We run autoreconf on ubuntu, and import files back on lacie:
autoreconf -vif
find -name .gitignore|xargs rm
git add -A
git checkout HEAD -- .gitignore m4/.gitignore src/win32/.gitignore test/.gitignore
git commit -m "add results of autoreconf -vif"
git branch -f master
  • Install dependencies, from log linked above. Note can use ipkg list to find package.
ipkg install openssl-dev librsync zlib
ipkg install gcc make automake autoconf libtool binutils autoconf
ipkg install optware-devel
  • Install uthash headers in /opt/include:
git clone https://github.com/troydhanson/uthash.git
cp uthash/src/ut* /opt/include/
  • OpenSSL error: unable to find OpenSLL library
./configure --prefix=/opt --sysconfdir=/opt/etc/burp --localstatedir=/opt/var
# checking For OpenSSL... no
# configure: error: Unable to find OpenSSL library
This is due to undefined reference:
grep -iC 10 openssl config.log|less
# /opt/arm-none-linux-gnueabi/lib/libdl.so.2: undefined reference to `_dl_tls_get_addr_soft@GLIBC_PRIVATE'
We relink some libraries as suggested here. Let's relink them all:
cd /opt/arm-none-linux-gnueabi/
mv lib lib.original
ln -sf /lib lib
cd -
  • Configure again, now fine:
./configure --prefix=/opt --sysconfdir=/opt/etc/burp --localstatedir=/opt/var
  • Make
make
# ... error about missing aclocal...

Digging further, we see that aclocal does not work because it requires Perl built with threads enabled. For this we need Perl 5.22 from Optware-ng

Build on Optware-ng

To install 

# Install dependencies
ipkg install openssl-dev librsync zlib
ipkg install gcc
ipkg install make automake autoconf libtool binutils autoconf
ipkg install optware-devel

# Install uthash headers in /opt/include:
git clone https://github.com/troydhanson/uthash.git
cp uthash/src/ut* /opt/include/

# Clone git repository
git clone https://github.com/grke/burp.git
cd burp
git checkout 2.1.6

# Generate the autoconf files
autoreconf -vif

# Configure
./configure --prefix=/opt --sysconfdir=/opt/etc/burp --localstatedir=/opt/var

# Make
make

# Install
make install
# Copied/created the following files/directories:
#   /opt/bin/vss_strip
#   /opt/sbin/burp
#   /opt/sbin/burp_ca
#   /opt/share/burp
#   /opt/share/doc/burp
#   /opt/share/man/man8/{bedup.8,bsigs.8,bsparse.8,burp.8,burp_ca.8,vss_strip.8}

# Install config
make install-configs
# Copied/created the following files/directories:
#     /opt/etc/burp

# If we want to create a .deb package:
# (Note: if fail, run make install install-configs w/o checkinstall first)
sudo checkinstall make install install-configs

Edit the following files:

  • /etc/burp/burp-server.conf (for backup server)
  • /etc/burp/burp.conf (for backup client)

For each client cname on the server (see cname in burp.conf), edit the file:

  • /etc/burp/clientconfdir/cname.

Start the server with: 

sudo burp -c /etc/burp/burp-server.conf

On the client, start a backup with: 

sudo burp -a b

Monitor ongoing backups with: 

sudo burp -s

Benchmarks

### On the server
egrep "(protocol|ratelimit|comp|ssl_ciphers).*=" /opt/etc/burp/burp-server.conf
pkill burp && rm -rf /opt/var/spool/burp/* && burp -c /opt/etc/burp/burp-server.conf

### On the client
egrep "(protocol|ratelimit|comp|ssl_ciphers).*=" /etc/burp/burp.conf
sudo burp -a b
/var/www/datasheet on zavcxl0005 -- no compression, protocol 2
du --apparent-size -BM -s /var/www/datasheet/
# 92M	/var/www/datasheet/
du -BM -s /var/www/datasheet/
# 92M	/var/www/datasheet/

### On the server
# protocol = 2
# compression = zlib0
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_compression = zlib0
sudo burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 22:35:07
#   End time: 2017-04-26 22:36:24
# Time taken: 01:17
#                          New   Changed Duplicate   Deleted     Total |  Scanned
#                    ------------------------------------------------------------
#              Files:        0         0         0         0         0 |       11
#        Directories:        0         0         0         0         0 |        6
#             Blocks:    14005         0     28811         0     42816 |        0
#        Grand total:    14005         0     28811         0     42816 |       17
#                    ------------------------------------------------------------
# 
#              Messages:             0
#              Warnings:             0
# 
#       Bytes estimated:      95598630 (91.17 MB)
#       Bytes in backup:             0
#        Bytes received:        109011 (106.46 KB)
#            Bytes sent:      96078512 (91.63 MB)
# --------------------------------------------------------------------------------
# 2017-04-26 22:36:24: burp[18005] End backup
# 2017-04-26 22:36:24: burp[18005] backup finished ok
  • CPU usage on server: 95% average.
  • transfer time: 77s.
  • transfer speed: 1.2MB/s.
/var/www/datasheet on zavcxl0005 - no compression, no encryption, protocol 2
### On the server
# protocol = 2
# compression = zlib0
# compression = zlib0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
sudo burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 22:52:53
#   End time: 2017-04-26 22:53:58
# Time taken: 01:05
#                          New   Changed Duplicate   Deleted     Total |  Scanned
#                    ------------------------------------------------------------
#              Files:        0         0         0         0         0 |       11
#        Directories:        0         0         0         0         0 |        6
#             Blocks:    14005         0     28023         0     42028 |        0
#        Grand total:    14005         0     28023         0     42028 |       17
#                    ------------------------------------------------------------
# 
#              Messages:             0
#              Warnings:             0
# 
#       Bytes estimated:      95598630 (91.17 MB)
#       Bytes in backup:             0
#        Bytes received:        109011 (106.46 KB)
#            Bytes sent:      96078512 (91.63 MB)
# --------------------------------------------------------------------------------
# 2017-04-26 22:53:58: burp[22408] End backup
# 2017-04-26 22:53:58: burp[22408] backup finished ok
  • CPU usage on server: 95% average.
  • transfer time: 65s.
  • transfer speed: 1.4MB/s.
### On the server
# protocol = 1
# compression = zlib0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 23:12:07
#   End time: 2017-04-26 23:12:23
# Time taken: 00:16
#                          New   Changed Duplicate   Deleted     Total |  Scanned
#                    ------------------------------------------------------------
#              Files:       11         0         0         0        11 |       11
#        Directories:        0         0         0         0         0 |        6
#        Grand total:       11         0         0         0        11 |       17
#                    ------------------------------------------------------------
# 
#              Messages:             0
#              Warnings:             0
# 
#       Bytes estimated:      95598630 (91.17 MB)
#       Bytes in backup:      95598630 (91.17 MB)
#        Bytes received:          1844 (1.80 KB)
#            Bytes sent:      95720309 (91.29 MB)
# --------------------------------------------------------------------------------
# 2017-04-26 23:12:23: burp[23561] Phase 2 end (send file data)
# 2017-04-26 23:12:23: burp[23561] backup finished ok
  • CPU usage on server: 95% average.
  • transfer time: 16s.
  • transfer speed: 5.7MB/s.
Increasing ratelimit
  • No effect on any protocol.
Protocol 2 - Backup again after duplicating all files
### On the server
# protocol = 2
# compression = zlib0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
sudo cp -r /var/www/datasheet/arm /var/www/datasheet/arm2    # We make a redundant copy inside
sudo burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 23:22:22
#   End time: 2017-04-26 23:22:35
# Time taken: 00:13
#                          New   Changed Duplicate   Deleted     Total |  Scanned
#                    ------------------------------------------------------------
#              Files:        0         0         0         0         0 |       22
#        Directories:        0         0         0         0         0 |       11
#             Blocks:        0         0     18024         0     18024 |        0
#        Grand total:        0         0     18024         0     18024 |       33
#                    ------------------------------------------------------------
# 
#              Messages:             0
#              Warnings:             0
# 
#       Bytes estimated:     191197260 (182.34 MB)
#       Bytes in backup:             0
#        Bytes received:          1156 (1.13 KB)
#            Bytes sent:        412007 (402.35 KB)
# --------------------------------------------------------------------------------
# 2017-04-26 23:22:35: burp[24000] End backup
# 2017-04-26 23:22:35: burp[24000] backup finished ok
  • CPU usage on server: 95% average.
  • transfer time: 13s.
  • transfer speed: 14.0MB/s.
Protocol 1 - Backup again after duplicating all files
### On the server
# protocol = 1
# compression = zlib0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
### On the client
# protocol = 0
# ssl_ciphers = eNULL
# ssl_compression = zlib0
sudo cp -r /var/www/datasheet/arm /var/www/datasheet/arm2    # We make a redundant copy inside
sudo burp -a b
# --------------------------------------------------------------------------------
# Start time: 2017-04-26 23:28:13
#   End time: 2017-04-26 23:28:30
# Time taken: 00:17
#                          New   Changed Duplicate   Deleted     Total |  Scanned
#                    ------------------------------------------------------------
#              Files:       11         0         0         0        11 |       22
#        Directories:        0         0         0         0         0 |       11
#        Grand total:       11         0         0         0        11 |       33
#                    ------------------------------------------------------------
# 
#              Messages:             0
#              Warnings:             0
# 
#       Bytes estimated:     191197260 (182.34 MB)
#       Bytes in backup:      95598630 (91.17 MB)
#        Bytes received:          1844 (1.80 KB)
#            Bytes sent:      95722406 (91.29 MB)
# --------------------------------------------------------------------------------
# 2017-04-26 23:28:30: burp[24200] Phase 2 end (send file data)
# 2017-04-26 23:28:30: burp[24200] backup finished ok
#
  • CPU usage on server: 95% average.
  • transfer time: 17s.
  • transfer speed: 10.7MB/s.

Troubleshoot

SSL connect error

sudo burp -a b
# 2017-04-25 23:09:47: burp[30776] SSL connect error
# 140585554515608:error:1409441B:SSL routines:ssl3_read_bytes:tlsv1 alert decrypt error:s3_pkt.c:1487:SSL alert number 51

This happens when using old certificates on backup client, typically after changing the configuration to point to another backup server [1]. 

cd /etc/burp
rm ssl_cert_ca.pem ssl_cert-client.key ssl_cert-client.pem
Retrieved from "https://miki.immie.org/mediawiki/index.php?title=Burp&oldid=9309"