Configuration LaCie-CloudBox: Difference between revisions

From miki
Jump to navigation Jump to search
Line 229: Line 229:
* Mount partition {{file|/dev/md6}} and copy back configuration changes (see above) from {{file|/dev/md6/1}} to {{file|/dev/md6/0}} (or vice-versa, depending on which image contains the new firmware).
* Mount partition {{file|/dev/md6}} and copy back configuration changes (see above) from {{file|/dev/md6/1}} to {{file|/dev/md6/0}} (or vice-versa, depending on which image contains the new firmware).
* Note that originally we tried to edit directly the partitions {{file|/dev/md4}} but this does not work since the configuration is overridden by files in {{file|/dev/md6}}.
* Note that originally we tried to edit directly the partitions {{file|/dev/md4}} but this does not work since the configuration is overridden by files in {{file|/dev/md6}}.

=== Install Optware ===
This is copied from the guide [http://lacie.nas-central.org/wiki/Category:CloudBox#Enabling_Secure_Shell]:
* Create a location for optware root:
<source lang="bash">
mkdir /shares/admin/opt # This assumes a share 'admin' exists
mkdir /opt
mount -o bind /shares/admin/opt /opt
</source>

* Manually download and Extract the ipkg-opt package.
:There's a chicken-and-egg situation we need to overcome, where we cannot install the ipkg-opt package without a package manager. We solve this by replicating the basic functions of the ipkg binary.

<source lang="bash">
cd /opt
feed=http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/stable/
feednative=http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/native/unstable
ipkg_name=`wget -qO- $feed/Packages | awk '/^Filename: ipkg-opt/ {print $2}'`
wget $feed/$ipkg_name
tar -xOvzf $ipkg_name ./data.tar.gz | tar -C / -xzvf -
rm $ipkg_name
</source>

* Configure Package Sources

<source lang="bash">
mkdir -p /opt/etc/ipkg
echo "src cross $feed" > /opt/etc/ipkg/feeds.conf
echo "src native $feednative" >> /opt/etc/ipkg/feeds.conf
</source>

* Prepare Root's Profile

<source lang="bash">
echo -e "\nexport PATH=/opt/bin:/opt/sbin:\$PATH" >> /root/.profile
source /root/.profile
</source>

* Update the Package Lists

<source lang="bash">
ipkg update
</source>

* Install the Optware Init Driver Script
:Create the file {{file|/opt/etc/rc.optware}} containing the following:

<source lang="bash">
#!/bin/sh

# Start all init scripts in /opt/etc/init.d
# executing them in numerical order.
#
for i in /opt/etc/init.d/S??* ;do

# Ignore dangling symlinks (if any).
[ ! -f "$i" ] && continue

case "$i" in
*.sh)
# Source shell script for speed.
(
trap - INT QUIT TSTP
set start
. $i
)
;;
*)
# No sh extension, so fork subprocess.
$i start
;;
esac
done
</source>

:Make the file executable:

<source lang="bash">
chmod 755 /opt/etc/rc.optware
</source>

* Install the Optware InitNG File

This file is used by initng to launch the {{file|rc.optware}} file we just created. Copy the contents into the file into {{file|/etc/initng/optware.i}}:

<source lang="bash">
#!/sbin/itype
# This is a i file, used by initng parsed by install_service

service optware {
need = unicorn/ready;
stdall = /var/log/messages;
script start = {
if test -z "${REAL_OPT_DIR}"; then
REAL_OPT_DIR=/shares/admin/opt/
fi
if test -n "${REAL_OPT_DIR}"; then
if ! grep ' /opt ' /proc/mounts >/dev/null 2>&1 ; then
mkdir -p /opt
mount -o bind ${REAL_OPT_DIR} /opt
fi
fi
[ -x /opt/etc/rc.optware ] && /opt/etc/rc.optware
};
script stop = {
umount /opt
};
}
</source>

* Add optware to the end of {{file|default.runlevel}}

<source lang="bash">
echo "optware" >> /etc/initng/runlevel/default.runlevel
</source>

* Tell initng to start Optware

<source lang="bash">
ngc --start optware
</source>

:After a reboot, /opt should be mounted, and any packages that install scripts in /etc/init.d/ (eg openssh) should have them executed at start-up time.


=== Install Git ===
=== Install Git ===

Revision as of 17:41, 24 April 2017

Reference information

uname -a
# Linux LaCie-CloudBox 2.6.31.14-svn7493 #1 Fri Oct 17 14:00:51 UTC 2014 armv5tel GNU/Linux
  • /proc/cpuinfo
cat /proc/cpuinfo
# Processor	: Feroceon 88FR131 rev 1 (v5l)
# BogoMIPS	: 999.42
# Features	: swp half thumb fastmult edsp 
# CPU implementer	: 0x56
# CPU architecture: 5TE
# CPU variant	: 0x2
# CPU part	: 0x131
# CPU revision	: 1
# 
# Hardware	: familybox
# Revision	: 0000
# Serial		: 0000000000000000
  • meminfo
cat meminfo
# MemTotal:         254348 kB
# MemFree:          115412 kB
# Buffers:            3944 kB
# Cached:            76468 kB
# SwapCached:            0 kB
# Active:            58464 kB
# Inactive:          63856 kB
# Active(anon):      42692 kB
# Inactive(anon):        0 kB
# Active(file):      15772 kB
# Inactive(file):    63856 kB
# Unevictable:           0 kB
# Mlocked:               0 kB
# SwapTotal:        262124 kB
# SwapFree:         262124 kB
# Dirty:                 0 kB
# Writeback:             0 kB
# AnonPages:         41932 kB
# Mapped:            19776 kB
# Slab:              11936 kB
# SReclaimable:       4772 kB
# SUnreclaim:         7164 kB
# PageTables:         1056 kB
# NFS_Unstable:          0 kB
# Bounce:                0 kB
# WritebackTmp:          0 kB
# CommitLimit:      389296 kB
# Committed_AS:     208068 kB
# VmallocTotal:     385024 kB
# VmallocUsed:         992 kB
# VmallocChunk:     382244 kB
  • free
free
#              total         used         free       shared      buffers
# Mem:        254348       138996       115352            0         3976
# -/+ buffers:             135020       119328
# Swap:       262124            0       262124

Mounting the drive in another pc, we see the following structure: 

sudo parted -l
# Model: ATA ST3000DM001-1CH1 (scsi)
# Disk /dev/sdb: 3001GB
# Sector size (logical/physical): 512B/4096B
# Partition Table: gpt
# 
# Number  Start   End     Size    File system     Name         Flags
#  1      1049kB  2097kB  1049kB                  grub_core    bios_grub
#  2      2097kB  203MB   201MB   ext2            boot_rescue
#  3      203MB   204MB   1049kB  ext2            nv_data
#  4      204MB   1815MB  1611MB  ext2            root_1       raid
#  5      1815MB  3426MB  1611MB  ext2            root_2       raid
#  6      3426MB  4499MB  1074MB  ext3            var          raid
#  7      4499MB  4768MB  268MB   linux-swap(v1)  swap         raid
#  8      4768MB  3001GB  2996GB                  user_data    raid
# 
# 
# Model: Linux Software RAID Array (md)
# Disk /dev/md4: 1611MB
# Sector size (logical/physical): 512B/4096B
# Partition Table: loop
# 
# Number  Start  End     Size    File system  Flags
#  1      0.00B  1611MB  1611MB  ext2
# 
# 
# Model: Linux Software RAID Array (md)
# Disk /dev/md5: 1611MB
# Sector size (logical/physical): 512B/4096B
# Partition Table: loop
# 
# Number  Start  End     Size    File system  Flags
#  1      0.00B  1611MB  1611MB  ext2
# 
# 
# Model: Linux Software RAID Array (md)
# Disk /dev/md6: 1074MB
# Sector size (logical/physical): 512B/4096B
# Partition Table: loop
# 
# Number  Start  End     Size    File system  Flags
#  1      0.00B  1074MB  1074MB  ext3
# 
# 
# Model: Linux Software RAID Array (md)
# Disk /dev/md7: 268MB
# Sector size (logical/physical): 512B/4096B
# Partition Table: loop
# 
# Number  Start  End    Size   File system     Flags
#  1      0.00B  268MB  268MB  linux-swap(v1)

The disk is organized as follows :

  • /dev/sdb4 and /dev/sdb5 are raid volumes that contains two copies of the same root partition. The other copy is used when doing a firmware update (ping-pong approach, see [1]). We mount them with:
mdadm --assemble --run /dev/md4 /dev/sdb4       # Done automatically on Ubuntu
mdadm --assemble --run /dev/md5 /dev/sdb5       # Done automatically on Ubuntu
mount /dev/md4 /mount/md4
mount /dev/md5 /mount/md5
  • /dev/sdb6 contains another raid partition, mounted as an additional layer over root partition above. It contains:
/dev/md6
    0/
        etc/
        root/
        var/
    1/
        etc/
        root/
        var/
    lost+found/
    tmp/

Configuration

First install

  • Network share available at smb://lacie-cloudbox/.
  • Cloudbox dashboard available at http://lacie-cloudbox.local.
  • setup smtp access (smtp.scarlet.be, port 25)
  • IP Configuration — fixed IP address 172.19.100.98
  • Enable the netbackup-share (port 873, encrypted port 22)
  • Create user beq06659, with private share.
MAC 00:d0:4b:96:34:32
IP 172.19.100.98
hostname lacie-cloudbox

Rooting

First rooting
  • Create telnetd.sh in a share root
cat > /smb/lacie-cloudbox/family/telnetd.sh <<- __END__
#!/bin/sh
/usr/sbin/telnetd -l /bin/sh
__END__
  • Get and build clunc:
# CLUNC Home: http://lacie-nas.org/doku.php?id=clunc
git clone http://git.lacie-nas.org/clunc.git
cd clunc
make
  • Run clunc, and reboot lacie:
./clunc -i 192.168.1.98
  • ...

When we get root access, we changed the following file:

  • Edit file /etc/initng/runlevel/default.runlevel:
-#sshd
+sshd
  • Edit file /etc/unicorn/unicorn_conf/unicorn.sharing.ssh.conf
-enabled: false
+enabled: true
  • Edit file /etc/ssh/sshd_config
-PermitRootLogin no
-AllowUsers netbackup
+PermitRootLogin yes
+AllowUsers root netbackup git
+Match User root
+Match User git
  • Add to file /etc/environment:
PATH=/opt/bin:/opt/sbin
  • Edit file /etc/hosts
  • Edit file /etc/lighttpd/lighttpd.conf
  • Edit file /etc/passwd (for user git)
Second rooting

After firmware upgrade, we need to root it again. This time clunc method no longer works.

  • Mount drive in another pc
  • Mount partition /dev/md6 and copy back configuration changes (see above) from /dev/md6/1 to /dev/md6/0 (or vice-versa, depending on which image contains the new firmware).
  • Note that originally we tried to edit directly the partitions /dev/md4 but this does not work since the configuration is overridden by files in /dev/md6.

Install Optware

This is copied from the guide [2]:

  • Create a location for optware root:
mkdir /shares/admin/opt                    # This assumes a share 'admin' exists
mkdir /opt
mount -o bind /shares/admin/opt /opt
  • Manually download and Extract the ipkg-opt package.
There's a chicken-and-egg situation we need to overcome, where we cannot install the ipkg-opt package without a package manager. We solve this by replicating the basic functions of the ipkg binary.
cd /opt
feed=http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/cross/stable/
feednative=http://ipkg.nslu2-linux.org/feeds/optware/cs08q1armel/native/unstable
ipkg_name=`wget -qO- $feed/Packages | awk '/^Filename: ipkg-opt/ {print $2}'`
wget $feed/$ipkg_name
tar -xOvzf $ipkg_name ./data.tar.gz | tar -C / -xzvf -
rm $ipkg_name
  • Configure Package Sources
mkdir -p /opt/etc/ipkg
echo "src cross $feed" > /opt/etc/ipkg/feeds.conf
echo "src native $feednative" >> /opt/etc/ipkg/feeds.conf
  • Prepare Root's Profile
echo -e "\nexport PATH=/opt/bin:/opt/sbin:\$PATH" >> /root/.profile
source /root/.profile
  • Update the Package Lists
ipkg update
  • Install the Optware Init Driver Script
Create the file /opt/etc/rc.optware containing the following:
#!/bin/sh

# Start all init scripts in /opt/etc/init.d
# executing them in numerical order.
#
for i in /opt/etc/init.d/S??* ;do

        # Ignore dangling symlinks (if any).
        [ ! -f "$i" ] && continue

        case "$i" in
           *.sh)
                # Source shell script for speed.
                (
                        trap - INT QUIT TSTP
                        set start
                        . $i
                )
                ;;
           *)
                # No sh extension, so fork subprocess.
                $i start
                ;;
        esac
done
Make the file executable:
chmod 755 /opt/etc/rc.optware
  • Install the Optware InitNG File

This file is used by initng to launch the rc.optware file we just created. Copy the contents into the file into /etc/initng/optware.i: 

#!/sbin/itype
# This is a i file, used by initng parsed by install_service

    service optware {
          need = unicorn/ready;
          stdall = /var/log/messages;
          script start = {
                  if test -z "${REAL_OPT_DIR}"; then
                      REAL_OPT_DIR=/shares/admin/opt/
                  fi
                  if test -n "${REAL_OPT_DIR}"; then
                      if ! grep ' /opt ' /proc/mounts >/dev/null 2>&1 ; then
                          mkdir -p /opt
                          mount -o bind ${REAL_OPT_DIR} /opt
                      fi
                  fi
                  [ -x /opt/etc/rc.optware ] && /opt/etc/rc.optware
          };
          script stop = {
                  umount /opt
          };
  }
  • Add optware to the end of default.runlevel
echo "optware" >> /etc/initng/runlevel/default.runlevel
  • Tell initng to start Optware
ngc --start optware
After a reboot, /opt should be mounted, and any packages that install scripts in /etc/init.d/ (eg openssh) should have them executed at start-up time.

Install Git

To enable git, we must:

  • Enable git logging in /etc/passwd.
  • Enable git ssh logging in /etc/ssh/sshd_config

Git needs a lot of memory when compressing its database (occurs at each fetch!). To limit this, run as root: 

git config --system pack.windowmemory 128m
git config --system pack.packsizelimit 1g
git config --system pack.threads 1
git config --system core.bigFileThreshold 16m

Then as user git, in all relevant projects (typically those with big files) 

cd ~/git/minecraft_nosaves.git
git config --local core.bigFileThreshold 10m

Install rsync

We do via user git, so:

  • Set /bin/bash as shell for user git in /etc/passwd
  • Set PermitUserEnvironment yes in /etc/ssh/sshd_config
  • Create file /shares/git/.ssh/environment to add rsync into path:
PATH=/opt/bin:/opt/sbin:/bin:/usr/bin:/sbin:/usr/sbin

Startup script

Create the file /opt/etc/init.d/S80config (chmod 755): 

#! /bin/sh

# Enable /bin/bash as shell for user git to also allow rsync + git in ssh login
sed -ri '/^git/{s_/home_/shares/git_; s_/bin/false_/bin/bash_}' /etc/passwd
# sed -ri '/^git/{s_/home_/shares/git_; s_/bin/false_/opt/bin/git-shell_}' /etc/passwd

# Change permission for /shares/git or sshd will complain (or add 'StrictModes off to /etc/ssh/ssh_config)
chmod go-w /shares/git

# Configure sshd
if ! grep -q git /etc/ssh/sshd_config; then 
    # Enable ssh login with user git
    sed -ri '/^AllowUsers/s/$/ git/;$aMatch User git' /etc/ssh/sshd_config
    # Enable user environment for user git (in order to set PATH)
    sed -ri $'/^PermitRootLogin/{a PermitUserEnvironment yes\n}' /etc/ssh/sshd_config
    # Restart sshd
    ngc --restart sshd
fi

# Add git to path
if ! grep -q /opt/bin /etc/environment; then echo "PATH=/opt/bin:/opt/sbin" >> /etc/environment; fi

# Add extra addresses to /etc/hosts
if ! grep -q "###hosts.inc###" /etc/hosts; then cat /opt/etc/hosts.inc >> /etc/hosts; fi

Create the hosts include file /opt/etc/hosts.inc: 

# ###hosts.inc### additional hosts for lacie-cloudbox
172.19.3.1      local.gateway            # Static
172.19.3.2      manticore                # Static?
172.19.100.1	gryphon
172.19.100.2	minimoy                  # DHCP Static        00:24:54:02:98:8d
172.19.100.3	unicorn                  # obsolete
172.19.100.4	beqleunxp1nb103          # obsolete
172.19.100.5	nxl67002ux               # obsolete
172.19.100.6	griffin                  # DHCP Static        68:b5:99:f2:4d:99
172.19.100.7	lambada                  # obsolete
172.19.100.8	reddragon
172.19.100.9    leviathan                # obsolete
172.19.100.10   nxl67063                 # obsolete
172.19.100.11   mandala                  # DHCP Static        00:1b:24:fa:a2:9b
172.19.100.12   nxl67002                 # obsolete
172.19.100.13   nxl67063ux               # obsolete
172.19.100.14   nxl67170ux               # DHCP Static        b8:ca:3a:cc:07:bf
172.19.100.15   pc-112-597               # DHCP Static        b8:ca:3a:93:91:27
172.19.100.16   zyratastic
172.19.100.17   graphicdaemon
172.19.100.18   xbmcbuntu                # DHCP Static        00:01:2e:4e:46:05 00:01:2e:4e:46:06
172.19.100.19   zavcxl0005               # DHCP Static        48:0f:cf:27:ad:ad
# 172.19.100.98	lacie-cloudbox           # Static
172.19.100.99	mnemosyne                # Static
172.19.100.102	minimoy-wlan             # DHCP Static        00:26:5e:37:83:ea
172.19.100.105	nxl67002ux-wlan          # DHCP Static
172.19.100.106	griffin-wlan             # DHCP Static        00:24:d7:7d:4e:94
172.19.100.107	apple-pi                 # Static
172.19.100.111  mandala-wlan             # DHCP Static        00:1d:92:13:d2:7d
172.19.100.114  nxl67170ux-wlan          # DHCP Static        84:3a:4b:22:0d:0c
172.19.100.116  zyratastic-wlan
172.19.100.118  xbmcbuntu-wlan           # DHCP Static        0c:d2:92:54:c1:53
172.19.100.119  zavcxl0005-wlan          # DHCP Static        ac:fd:ce:40:c5:31
172.19.100.120	galaxy

Install Unison

  • ipkg provides unison 2.27.57. It can be installed via ipkg install unison.
ipkg list | grep unison
# unison - 2.27.57-2 - A cross-platform file-synchronization tool.
  • To get a more recent version, see Unison to compile from source. v2.40.102 is the same as on Ubuntu Trusty 14.04.

Netconsole server

On lacie-cloudbox:

  • First install netcat:
ipkg install netcat
  • Then start the server:
mkdir -p /shares/beq06659/netconsole
nc -l -p 6666 -u >/shares/beq06659/netconsole/zavcxl0005-netconsole&

On the client, at reboot/wake-up: 

NETCON_TGTHOST=lacie-cloudbox
NETCON_PORT=6666
echo "[------------] Redocking on $(date +"%Y-%m-%d %H:%M:%S")..." | nc -w 1 -u -p 6665 $NETCON_TGTHOST $NETCON_PORT
if [ -n "$NETCON_TGTHOST" ]; then 
    NETCON_TGTIP=$(getent hosts $NETCON_TGTHOST|awk '{print $1}')
    ping -c 1 $NETCON_TGTIP >/dev/null
    NETCON_TGTMAC=$(arp -n $NETCON_TGTIP|awk '/ether/{print $3}')
    echo "Setting up netconsole for location '$LOCATION' ($NETCON_TGTHOST,$NETCON_TGTIP/$NETCON_TGTMAC)"
    sudo rmmod netconsole 2> /dev/null
    sudo modprobe netconsole netconsole=@/,$NETCON_PORT@$NETCON_TGTIP/$NETCON_TGTMAC
else 
    echo "netconsole - Unknown location $LOCATION"
fi

Shares

family
  • Mount with
sudo mount //lacie-cloudbox/beq06659 /mnt/any -o noperm,iocharset=utf8,sec=ntlm,username=public,password=


beq06659
  • Mount with
sudo mount //lacie-cloudbox/beq06659 /mnt/any -o noperm,iocharset=utf8,sec=ntlm,username=beq06659

Backup

Backup Mnemosyne

We use unison.

BackupPC

Fix broken pipe errors

Backuppc requires lot of memory (rsync). To free as much as possible, I kill all python stuff (except ublock-daemon) and all media stuff: 

pkill -SIGSTOP -f unicorn            # I stop it first so that it can't react
pkill -SIGKILL -f unicorn            # then kill it
sleep 1 && pkill -SIGSTOP -f unicorn # then stop it again because it respawns
pkill -SIGSTOP -f mt_daapd
pkill -SIGSTOP twonky
pkill -SIGSTOP php-cgi
pkill -SIGKILL -f mt-daapd           # media server stuff
pkill -SIGKILL twonky                # media server stuff
pkill -SIGKILL php-cgi               # php-cgi server launcher by lighttpd

Duplicity (Obsolete)

This section is OBSOLETE — I no longer use duplicity


  • Backup done with deja-dup, which use duplicity as back-end
  • Duplicity can't access samba share. Instead mount samba share locally, and use file:///smb/lacie-cloudbox/sharename instead. Note the triple slash.
  • View available backup:
duplicity collection-status file:///smb/lacie-cloudbox/beq06659
  • List file in current backup
duplicity list-current-files file:///smb/lacie-cloudbox/beq06659

Metadata information is stored in ~/.cache/duplicity. Attention, it seems this can grow quite large.

DHCP Server

  • install
# Install the service
ipkg install dhcp
mkdir /opt/var/run
# Edit the configuration
vi /opt/etc/dhcpd.conf
# Restart the service
/opt/etc/init.d/S56dhcpd
  • DHCP daemon configuration: /opt/etc/dhcpd.conf
  • Current leases are at /opt/etc/dhcpd.leases
  • Now I disabled the dhcpd, and will try again the one on vDSL+ router:
cp /opt/etc/dhcpd.conf /opt/etc/dhcpd.conf.bak     # Backup config if we want to reinstall
ipkg remove dhcp
  • DHCP server on the VDSL+ router is a huge pile of sh*te, so I enable it again:
ipkg install dhcp
cp /opt/etc/dhcpd.conf.bak /opt/etc/dhcpd.conf
/opt/etc/init.d/S56dhcpd

To do

  • Current backup only includes files that are readable by user beq06659. It should include all files.
  • Exclude some unnecessary folders. Organize these folders to ease future backup.
  • Backup / filesystem (at least /etc, /usr/local)
Backup solutions
  • deja-dup
  • Need one setup for each pc
  • No centralized management. Backup job could happen all at the same time. Same files between pc are stored multiple times.
  • Easy to setup
  • How to backup root filesystem?
  • duplicity
  • backuppc
  • How to install?
  • Will the NAS have enough memory?
  • bacula
  • Package readily available in OptWare
  • rsnapshot

See question on bacula/backuppc/rsnapshot at http://serverfault.com/questions/282228/bacula-vs-backuppc

Rooting NAS and install new app
  • TBC

Troubleshoot

sshd

  • See log at /var/log/messages
  • Check sshd config /etc/ssh/sshd_config:
  • Make sure user is authorized:
 AllowUsers root netbackup git
  • If user home is on /shares mount, then disable StrictModes
StrictModes no
  • Enable user environment
PermitUserEnvironment yes
then
env > ~/.ssh/environment
vi ~/.ssh/environment                # Remove useless variables
  • Edit /etc/passwd:
git:x:503:100:None:/shares/git:/opt/bin/git-shell
  • Restart sshd:
ngc --restart sshd

git

  • Check that environment is ok [3]
ssh git@lacie-cloudbox  "which git-upload-pack"

ntp

  • Box loses regularly time synchronization, which then affects backuppc
  • Workaround: log into https://lacie-cloudbox/, and reset ntp (uncheck / check ntp server).

slow performance, lot of swapping

The Lacie-Cloudbox only has 256MB physical memory, and some processes takes a lot of memory. When doing some unusual tasks, we can disable (stop) temporarily these processes, without apparently damage: 

pkill -SIGSTOP twonky        # Usually consuming around 100MB+
pkill -f -SIGSTOP unicorn    # A python process running /usr/lib/unicorn/run.py (and consuming 100MB+)

To restart: 

pkill -SIGCONT twonky
pkill -f -SIGCONT unicorn
Retrieved from "https://miki.immie.org/mediawiki/index.php?title=Configuration_LaCie-CloudBox&oldid=9276"