Linux Disk Management

Related pages

• Grub
• NTFS
• NFS
• ReiserFS
• Samba

SSD Management

See SSD Tuning for Linux.

Devices and Partitions

Some GUI software:

• gparted

Some CLI software:

• fdisk
• sfdisk
• parted
• gdisk (to deal with new GPT partition, see this link from microsoft for more info)

Typically, to view all devices and partitions:

sudo fdisk -l                                       # View ALL devices and partitions
sudo sfdisk -l                                      # idem

Some examples:

$ sudo fdisk -l /dev/sda                            # Show partition table for device /dev/sda
$ sudo fdisk -l -u /dev/sda                         # ... using sector as unit
$ sudo parted -l                                    # Show partition table of all devices
$ sudo parted /dev/sda print                        # ... of only device /dev/sda
$ sudo parted /dev/sda unit cyl print               #     ... using cylinder as unit
$ sudo parted /dev/sda unit s print                 #     ... using sector as unit (more accurate)
$ sudo sfdisk -l -uS /dev/sda                       # Show partition table for device /dev/sda
$ sudo sfdisk -d /dev/sda >sda-sfdisk.dump          # Dump partition in a format that can be understood by sfdisk
$ sudo sfdisk /dev/sda <sda-sfdisk.dump             # Restore a dumped partition table
$ sudo dd if=/dev/sda of=sda.mbr bs=512 count=1     # Save the complete MBR (table + boot code)

Use partprobe to force the kernel to re-read the MBR (re-read the partition table, see [1]). Or alternatively one can use fdisk to re-rewrite the same partition and force a re-read. And that are more solutions too ([2]):

$ sudo partprobe
# Or use fdisk
$ sudo fdisk /dev/sda
Command: v
Command: w
# Or use blockdev
$ sudo /sbin/blockdev --rereadpt /dev/hda
# Or use sfdisk
$ sudo sfdisk -R /dev/sda

UUID and labels

• Persistent block device naming
• UUID et Label

Run sudo blkid to get the UUID number.

blkid
# /dev/sda1: LABEL="AWS_System" UUID="023C4FC93C4FB687" TYPE="ntfs" 
# /dev/sda2: LABEL="BDEdrive" UUID="7C53861201698F3D" TYPE="ntfs" 
# /dev/sda3: LABEL="BOOT" UUID="0af7ef1a-cf55-4e67-913f-e53711178a70" TYPE="ext3" 
# /dev/sda5: UUID="754ca35b-fe65-4fce-a06d-8197f9494d7a" TYPE="reiserfs"

sudo lsblk -f shows a graphical representation (not for GPT system though):

$ sudo lsblk -f
NAME                FSTYPE   LABEL      MOUNTPOINT
sda                                     
├─sda1              ntfs     AWS_System /c
├─sda2              ntfs     BDEdrive   
├─sda3              ext3     BOOT       /boot
└─sda4

Note:

• blkid shows the result of last execution by root. If you created/removed partitions, do:

sudo blkid -g                  # Remove devices that no longer exist
sudo blkid                     # Update uuid cache & show the uuid list

• Alternatively, list /dev/disk/by-uuid/ or /dev/disk/by-label/:

ls -l /dev/disk/by-uuid
# total 0
# lrwxrwxrwx 1 root root 10 Jul  8 16:20 023C4FC93C4FB687 -> ../../sda1
# lrwxrwxrwx 1 root root 10 Jul  8 16:20 0af7ef1a-cf55-4e67-913f-e53711178a70 -> ../../sda3
# lrwxrwxrwx 1 root root 10 Jul  8 16:20 754ca35b-fe65-4fce-a06d-8197f9494d7a -> ../../sda5
# lrwxrwxrwx 1 root root 10 Jul 12 17:56 7C53861201698F3D -> ../../sda2

• To change UUID of ext filesystem:

tune2fs /dev/{device} -U {uuid}           # See man tune2fs for options

• On GPT systems, you can view the GUID under linux with

sudo sgdisk -i 1 /dev/sda
# Partition GUID code: C12A7328-F81F-11D2-BA4B-00A0C93EC93B (EFI System)
# Partition unique GUID: 2C47C282-EE6E-45DE-A5AD-E8658CA67DE6
# First sector: 2048 (at 1024.0 KiB)
# Last sector: 390625 (at 190.7 MiB)
# Partition size: 388578 sectors (189.7 MiB)
# Attribute flags: 1000000000000000
# Partition name: 'EFI System'

GUID is set with

sudo sgdisk -u 1:2C47C282-EE6E-45DE-A5AD-E8658CA67DE6 /dev/sda

GPT, EFI, MS reserved partition

• Windows and GPT FAQ
• Frequently asked questions about the GUID Partitioning Table disk architecture

GUID Partition Table

The GUID Partition Table (GPT) is a new partition scheme that replaces the legacy scheme called MBR.

• The GPT usually has a protective MBR, which is a legacy MBR sector with a single partition (code OxEE) that spans the whole disk (or as much as possible)
• GPT imposes no limit on the number of partition (but currently limited to 128 on Windows).
• Partitions in the GPT are identified via their GUID

How does the GUID in the GPT relates to the one in the partition itself, like the one set by tune2fs -U <uuid>?

EFI System Partition

• The EFI System Partition (ESP) contains all the files that are necessary for booting the operating system
• It is usually 100MB in size.
• It has a specific GUID

DEFINE_GUID (PARTITION_SYSTEM_GUID, 0xC12A7328L, 0xF81F, 0x11D2, 0xBA, 0x4B, 0x00, 0xA0, 0xC9, 0x3E, 0xC9, 0x3B)

Microsoft Specific Partition

• Reserved for future use by Windows in case some extra is needed (for instance dynamic disks). When so, the partition would be reduced and a new partition is created. This is to avoid using hidden sectors.
• Contains no relevant information.
• It has a specific GUID

DEFINE_GUID (PARTITION_MSFT_RESERVED_GUID, 0xE3C9E316L, 0x0B5C, 0x4DB8, 0x81, 0x7D, 0xF9, 0x2D, 0xF0, 0x02, 0x15, 0xAE)

Resizing Partitions

gparted

Probably one of the best way to edit/resize/move partition is to use the GUI tool gparted. It suports many different file systems, and allows for both resizing the file system but also updating the partition table.

If no GUI is available, here a few recipes for command-line.

Reiserfs

1. Use resize_reiserfs to resize the partition, and get the new partition size

resize_reiserfs -s -4G /dev/sda6               #Must be unmount
df

2. Change the partition table

sudo sfdisk -d /dev/sda >sda-sfdisk.dump          # Edit sda-sfdisk.dump

3. Run reiserfsck

sudo reiserfsck --rebuild-sb
sudo reiserfsck --fix-fixable

Mounting Partitions

See also reference pages above

Using /etc/fstab

# NTFS
UUID=XXXXXXXXXXXXXXXXXXXXX /media/windows ntfs defaults,umask=007,gid=46 0 1

Partitions can then be mounted with mount <mount-point>

Using mount

# NTFS - mount point /media/windows must be chgrp plugdev
sudo mount -t ntfs -o defaults,umask=007,gid=46 /dev/sda1 /media/windows
# SAMBA
sudo mount -t cifs -o username=baddreams,uid=1000,gid=124 //phoenix/D$ /net/phoenix/d

Remounting root partition read-write

If /etc/fstab is corrupted, boot process might stop while root partition is mounted read-only. To remount it in read-write mode in order to fix /etc/fstab (see [3]):

mount -n -o remount,defaults /dev/sda1 /       # -n means do not update /etc/mtab (when /etc is ro)

Boost ext3/4 performance by enabling data writeback and disabling atime

Data writeback leads to faster performance on ext3/4 filesystem, at the cost of possible loss of new data in case of system crash (old data magically reappear) (see [4]). To enable it simply add data=writeback to mount options in /etc/fstab. Also disable update of atime (access time):

/dev/hda1 / ext3 defaults,errors=remount-ro,noatime,data=writeback 0 1

Unmount partition first! Either unmount the partition, or first run tune2fs to update the current mount flag:

tune2fs -o journal_data_writeback /dev/sda1

Backup

References

• [1] — A comprehensive analysis Backing up Linux and other Unix(-like) systems
  Recommends DAR. tar, rsync, rdiff-backup are also options
• [2] — [5] for using cpio in order to preserve hardlinks

cp

cp can preserve all metadata, ownership, permissions, etc, as long as the user has the necessary rights and metadata are supported by the destination file system [6].

cp -a src dst              # GNU cp -a copies recursively preserving as much structure and metadata as possible.
sudo cp -a src dst         # ... running as root to preserve ownership

rsync

rsync can preserve all metadata, ownership, permissions, etc, as long as the user has the necessary rights and metadata are supported by the destination file system [7].

Advantages over cp ([8]):

• Only copy updated parts of an updated file (handy for incremental copies)
• has a --delete option
• Use encryption / decryption (handy over network)

rsync -a src dst           # -a, --archive         archive mode; equals -rlptgoD (no -H,-A,-X)
rsync -aH src dst          # ... -H, --hard-links  preserve hard links
rsync -aHA src dst         # ... -A, --acls        preserve ACLs (implies -p)
rsync -aHAX src dst        # ... -X, --xattrs      preserve extended attributes

Pro	Con
partition to partition cloning network support (using ssh). file-level backup, so can copy to a different filesystem type and size.	not a single archive. metadata stored as metadata in destination filesystem. complex set of options.

rsync goal is to synchronize 2 remote file systems over the network

My set of command line options (sudo pre-activation credits to [(credit http://crashingdaily.wordpress.com/2007/06/29/rsync-and-sudo-over-ssh/)] and [9])

#If needed, pre-activate sudo on remote system. Flag -t required to solve 'sudo: no tty present and no askpass program specified'
#
# Also, this requires the following line in /etc/sudoers:
#
#    Defaults     !tty_tickets
#
stty -echo; ssh -t user@server sudo -v; stty echo

sudo rsync -aHAXS --delete --rsync-path "sudo rsync" --numeric-ids -h -v --exclude='lost+found' user@server:/remote/path  /local/dest
# -a, --archive           aka. preserve almost everything (equiv. to -rlptgoD, i.e. --recursive, --links, --perms, --times, 
#                         --group, --owner, --devices, --specials)
# -H, --hard-links        preserve hardlinks
# -A, --acls              preserve ACLs (implies --perms)
# -X, --xattrs            preserve extended attributes
# -S, --sparse            handle sparse file efficiently
# --delete                delete extraneous files from the receiving side
# --rsync-path            command executed on remote system
# --numeric-ids           use gid / uid instead of user/group name for file permissions
# -v, --verbose           display file while transfering
# --exclude='lost+found'  useful on ext3/ext4

Some options to consider adding:

# -z, --compress          might increase txf speed on slow network (internet)
# -h, --human-readable
# --stats
# -P                      equiv. to --partial --progress (quite verbose)
# -v -v                   more verbose

tar

tar can preserve all metadata, ownership, permissions, etc, as long as the user has the necessary rights and metadata are supported by the destination file system [10].

(cd src;tar cf - .) | (mkdir dst;cd dst;tar xf -)    # create src as before
tar cf - . | tar xCf directory -                     # Same

pax (POSIX tar)

pax can preserve all metadata, ownership, permissions, etc, as long as the user has the necessary rights and metadata are supported by the destination file system [11].

mkdir dst
pax -rw src dst          # Same as tar, but pack and unpack in a single process

BackupPC

Tools used by Phil Teuwen. See Backuppc.

CloneZilla

Pro	Con
	Not a standalone program (only bootable cd / usb)

cpio

Some standard tool (see also [2] above).

DAR

Pro	Con
Single archive (containing all metadata)	transfer over network not easy. No immediate support for pipes (like ntfsclone)

DAR is recommended by [1] above. I personally tried the transfer through netword capability, but without success (broken image)

dump / restore

Backup tool for ext2/ext3 (/ext4 ?). See Backup or snaphot tool for ext4, but requires LVM2 for snapshot.

FSArchiver

Pro	Con
	* Does not support archiving through network (pipe). So one cannot save a partition, and restore it immediately on another machine through network for instance.

See this tutorial.

fsarchiver -v savefs /mnt/backupdrive/my-backup.fsa /dev/sda4
fsarchiver restfs -v /mnt/backupdrive/my-backup.fsa id=0,dest=/dev/sda4
sudo mount -o remount,ro /dev/sda4                                         # To remount read-only if complain it is mounted already

ntfsclone

Pro	Con
Simple and fast output/input through pipes (for compression, network transfer) partition-2-partition cloning

Simply the best for ntfs backup (support partition-2-partition backup through network).

ntfsclone

Pro	Con
Single archive	No support for pipes (compression, transfer via network) Destination partition must be same size as source partition Lack support for some FS (partimage does not support ext4, partclone does not support reiserfs, despites what manual says)

PartImage is another solution, but it does not support ext4.

RAMFS / TMPFS

References:

• http://www.thegeekstuff.com/2008/11/overview-of-ramfs-and-tmpfs-on-linux/
• http://en.wikipedia.org/wiki/Tmpfs

Using RAMFS and TMPFS you can allocate part of the physical memory to be used as a partition. This partition can be mounted as a regular hard disk partition to accelerate tasks that requires heavy disk access (this partition could store for instance a database, or a version control repository...)

Access Control

References:

• Part 1: How to work with Access Control Lists from the Command Line
• Part 2: How to work with Access Control Lists from the Command Line
• Using SGID to Control Group Ownership of Directories

Using SGID bit to Control Group Ownership

SGID bit allows for controlling the Group Ownership of files within a directory:

mkdir /data/testacl
chgrp git /data/testacl                # Set group to 'git'
chmod g+s /data/testacl                # Set SGID bit
cd /data/testacl
touch file                             # Now 'file' has group 'git', independently of current user primary group

This is nice, but access condition is still dependent on user's umask setting. Also, moving or copying files ignore the sticky bit.

Using ACL to set default access control

ACL must be installed:

sudo apt-get install acl

... and enabled on the target file system in /etc/fstab:

/dev/sda7    /data     ext4      defaults,acl     0     2

Now, let's say that default permission is 'rwx' for file created in our 'test' directory above:

cd
setfacl -m d:group:git:rwx /data/testacl  # By default, all members of group 'git' will have rwx access
                                          # Independently of user's umask setting
umask 022
touch /data/testacl/file022               # File 'file022' is still writable for group 'git'

However this does not work if files are copied or moved into the directory. In that case, files may either lose the group access flags, or even lose group ownership (see [12] for more). This could be a problem if for instance some application is unpacking some files in a temporary directory and then moves them to our ACL-controlled directory.

Change session primary group

We can change the primary group of the current session (and all sub-processes) so that any files created in the session belongs to some given group. This method is robust against moving / copying files into a directory, as long as these files have created in the same session. As a drawback however, it requires to first run a command to do the group switch:

See commands sg or newgrp.

efs attributes

See command lsattr and chattr (for instance the i, immutable, attribute).

Secure Wipe

Easiest and fastest method, use shred with one random pass and one zero pass (from []). This is safe enough according to this article:

sudo shred -v -z -n 1 /dev/sda