Linux Commands

File System

dd

Convert and copy a file.

dd returns current copy status when sent an HUP signal. Type in another shell:

while :; sleep 5; do killall -s SIGHUP1 dd; done

This will force dd to update its status every 5sec.

find

See here for further examples on how to combine find and xargs.

Frequent pitfall! The semi-colon ; must be escaped in Bash !

find . -name "*.jpg"                                     # find files/directories whose name matches specified pattern
find . -path "*/cur/*"                                   # find files/directories whose path spec matches specified pattern
find . -type d -name "cur"                               # find only directories whose name matches specified pattern
find . -exec echo '{}' \;                                # semi-colon must be escaped!
find . -exec echo one '{}' \; -exec echo two '{}' \;     # to execute several commands on one match
find . -name \*.jpg | xargs echo                         # much faster version using xargs
find . -name \*.jpg | xargs -n 2 echo                    # limit #args passed to echo to 2 max.
find . -name \*.jpg -print0 | xargs -0 ls                # Use NULL separator if files contains space or special char !!!
find . -name 'dirname' -prune -o -iname '*.txt' -print   # Find all text files, but exclude any directory named dirname
find . -path './path' -prune -false -o -iname '*.txt' -print
                                                         # Find all text files, but exclude path ./path
find . -iname '*.zip' -exec sh -c 'unzip -l "$0" | egrep -o " [0-9]+ file"' '{}' \;
                                                         # How to use pipe in -exec.

grep

grep -Rsl PATTERN [FILE]            # Recursive, no error output, only list filename
grep BASIC-REG-EXP-PATTERN [FILE]   # Use classic regexp (like "dma\|DMA")
egrep EXT-REG-EXP-PATTERN [FILE]    # Same as grep -E. Use extended regexp (like "dma|DMA")
fgrep FIXED-STRINGS-REG-EXP [FILE]  # Same as grep -F. Pattern is a list of strings to match.
grep -n PATTERN [FILE]              # Print matched line numbers.
grep -- "-s" [FILE]                 # Search for text "-s"
grep -e "-s" [FILE]                 # Search for text "-s" - alternative solution
grep -R -include=*.in PATTERN *     # Search recursively through folders, limiting to files matching pattern "*.in"
grep -R PATTERN *.in                # Idem, but matching pattern "*.in" also applies to folders.
grep -o PATTERN [FILE]              # (--only-matching) Print only the matched (non-empty) parts of a matching line.

tar

tar -cvzf archive.tgz somedir/      # Recursively archive & compress the directory somedir/
tar -cvzf archive.tgz --exclude=notme somedir/
                                    # Same as above, but exclude any file or directory named notme in the hierarchy
                                    # (this can potentially exclude several files/directory)
tar -xvzf archive.tgz --exclude=somedir/not/me somedir/
                                    # Same as above, but exclude specifically the file or directory at a specific location
tar -xvzf archive.tgz               # Extract the content of compressed archive archive.tgz (will create somedir/)
tar -tvzf archive.tgz               # List the content of compressed archive archive.tgz (equiv. of tar --list -vzf ...)

Miscellaneous

badblocks

An utility to detect bad sectors on USB flash drive or so.

dosfslabel

Change FAT16/FAT32 disk label

sudo dosfslabel /dev/sdc1 UBUNTULIVE

install-mbr

Install MS-like Master Boot Record (see [1])

sudo apt-get install mbr
install-mbr -i n -p D -t 0 /dev/sda        # Fix MBR on /dev/sda
                                           # -i interrupt n = none (do not display a MBR prompt)
                                           # -p partition D = boot the partition with the bootable flag set
                                           # -t timeout 0 = do not wait to boot
install-mbr -i n -p D -t 0 opensource.mbr  # Create a file opensource.mbr containing the generated MBR

mkfs

Format disk utility

sudo mkfs.vfat -F 32 -c /dev/sdc1

ms-sys

Install MS-compatible Master Boot Record (similar to FIXMBR on WinXP rescue console, see [2])

# Get ms-sys from http://ms-sys.sourceforge.net/#Download
sudo apt-get install gettext
make
sudo make install
sudo ms-sys -m /dev/sdb

relocntfs

relocntfs (see [3]) is more or less the equivalent of the FIXBOOT command in the Windows Recovery Console. It is among other available on the Trinity Rescue Kit. For instance, to fix the boot record of a windows partition at /dev/sda2 issue the commands:

relocntfs /dev/sda2        
relocntfs -w /dev/sda2                    # To actually write the new boot record

stat

display file or file system status

tree

List the contents of directories in a tree-like format.

Network

autossh

autossh - Automatically restart SSH sessions and tunnels.

autossh by default monitors the ssh connection through a dedicated port to see whether the current ssh connection must be restarted. However the simpler is to use the ssh config option ServerAliveCountMax and ServerAliveInterval so that ssh exits automatically when the connection is down. In that case, autossh will then restart ssh automatically without need of additional monitoring.

Add to your ~/.ssh/config:

Host *
  ServerAliveCountMax       3    # default value actually
  ServerAliveInterval       15   # ssh will exit after 3 x 15s = 45s

Example of uses:

$ autossh -M 0 -f -D 1080 noekeon       # -M 0 tells autossh to only restart ssh upon ssh's exit

dig

dig stands for domain information groper. It replaces the deprecated nslookup. It comes with package dnsutils:

sudo apt-get install dnsutils

multitee

multitee sends multiple inputs to multiple outputs.
Check this page.
Original is here, or can also be found here on Debian.

Here a patch to build it on Cygwin. The patch ports calls to BSD signal.h API (struct sigvec, sigvec(), sigmask(), sigblock()...) to the POSIX API (struct sigaction, sigaction(), sigprocmask(), sigfillset()...):

$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin

Example of use:

$ multitee 0-1,4,5 4> foo 5> bar             # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7                 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0   # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1                           # same as 'socat - FD:3'

netcat

TCP-IP swiss army knife (equivalent of the telnet program. Check wikipedia:netcat. Also known as command nc).

netstat

Print network connections, routing tables, interface statistics, masqurade connections, and multicast memberships

netstat -utpn      #Active ports, tcp, socket program PID, numeric
netstat -lutpn     #Listen ports, tcp, socket program PID, numeric
netstat -autpn     #All (active and listen), tcp, socket program PID, numeric
netstat -rn        #Kernel route table, numeric

When listing sockets (default output), you'll get an output like:

% netstat -at Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 *:time *:* LISTEN tcp 0 0 localhost:mysql *:* LISTEN tcp 0 0 andLinux.local:43449 windows-host:x11 ESTABLISHED

% netstat -atn Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:37 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN tcp 0 0 192.168.11.150:43449 192.168.11.1:6000 ESTABLISHED

Local Address

* or 0.0.0.0 means that the process accepts connection from any interface.

127.0.0.1 means it only accepts connection on localhost loopback (and so only connection that originates from local PC as well).

Any other IP address means that the process listen on the given port at the given IP address

/etc/init.d/nscd

This is not really a command, but the init service Name Service Cache Daemon.

sudo /etc/init.d/nscd restart                # To restart daemon and flush DNS cache

socat

Command-line utility that establishes two bidirectional byte streams and transfers data between them ([4]). socat is the more powerful version of netcat. Check the homepage. And also this page on Yobi for examples on how to use socat to bypass a proxy.

socat can easily tunnel connections:

socat -ly 'TCP4-LISTEN:143,reuseaddr,fork' PROXY:ton.imap.server:143|TCP:134.27.168.36:8080

socat can also be used as SSH ProxyCommand:

ProxyCommand socat -ly - 'PROXY:%h:%p|TCP:proxy.server:8080'
# Using socat v2.0.0 beta
ProxyCommand socat -ly - 'PROXY:%h:%p,proxyauth=user:pass|SSL,verify=0|PROXY:my.server:443,proxyauth=user:pass|TCP:big.brother.proxy:8080'

socat can be easily used as a replacement of telnet:

socat tcp:<host>:<port> -       # <port> can be a port number or service name (telnet,imap...)

Some useful command-line options:

socat -ly                       # Writes messages to syslog instead of stderr; severity as defined with option -d
socat -v                        # Writes the transferred data to their target stream but also to stderr (text mode)

QUESTION TO MR DOEGOX:
- WHY -s in SED?
- Where does it get info on % for socat?

$ socat echo -
$ socat echo STDIO
$ socat echo STDOUT%STDIN
$ socat echo STDIO%STDIO
$ socat echo -%-
$ socat echo FD:1%FD:0
$ socat echo 1%0

/usr/local/bin/socat TCP:127.0.0.1:1234 -%EXEC:"awk 'BEGIN{print \"BANNER\";fflush()}/BANNER/{next}//{print;fflush()}'"

Testing socat:
- /usr/local/bin/socat - TCP-LISTEN:1234
- /usr/local/bin/socat TCP:127.0.0.1:1234 EXEC:"/bin/sed -s 's/foo/bar/g'"%EXEC:"/bin/sed -u 's/toto/tata/g'"

  First open a port for listening and connect it to STDIO
  In another shell, connect and do the black magic...


reference:
- http://eros.ph0k.eu/~nizox/twin/V2/modules/socat-1.6.0.0/EXAMPLES
- http://www.rschulz.eu/2008/09/ssh-proxycommand-without-netcat.html
		$ exec 3<>/dev/tcp/zeus.yobi.be/22;(cat <&3 & );cat >&3
		SSH-2.0-OpenSSH_5.1p1 Debian-3
- http://thesmithfam.org/blog/2006/05/23/bash-socket-programming-with-devtcp-2/
- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464
- http://www.mtu.net/~engstrom/ssh-proxy.php



ATTEMPT 3:
Window 1: socat -v -%'EXEC:/home/beq06659/.ssh/mys2.sh' TCP-LISTEN:1234
Window 2: /usr/local/bin/socat -ly -v -%"EXEC:sed -unf /home/beq06659/.ssh/sedy" "PROXY:ftp.noekeon.org:22|TCP:localhost:1234"

File mys2.sh:
	#! /bin/bash
	echo -e "HTTP/1.1 200 Connection established\r\n\r\n"
	cat
File sedy:
	1 s/^/SSH-2.0-OpenSSH_5.1\n/p; /SSH-2.0-OpenSSH_5.1/d; /SSH-2.0-OpenSSH_5.1/! p

LESSONS LEARNED:
- READ THE CHANGES DOC, especially for BETA SW
- SOCKET = bidir --> a filter = 4 descriptors
- PIPE = unidir --> a filter = 2 descriptor (stdin/stdout)

Package Management

dpkg

Package manager for Debian / Ubuntu. See Package Management#Ubuntu / Debian

rpm

See Package Management#RPM's

System

Sudo / Gksudo

See also Ubuntu's page at [linux reference script security sudo ubuntu].

ls | sudo tee /root/somefile                # Instead of redirection sudo ls > /root/somefile
ls | sudo tee -a /root/somefile             # ... same but append instead
sudo sh -c "ls > /root/somefile"            # Yet another way to do redirection with sudo
sudo !!                                     # Repeat last command with root privilege
sudo -i                                     # Get a root *login* shell
sudo su -                                   # ... another solution
sudo -s                                     # Root shell but keeping current environment
sudo su                                     # ... another solution
sudo -k                                     # Reset sudo password timeout

gksudo is the graphical equivalent of sudo:

gksudo gedit /etc/fstab

Showkey

showkey examine the codes sent by the keyboard (linux kernel only - for X keyboard events, see xev!).

showkey -k                                  # Dump key codes (default)
showkey -s                                  # Dump scan codes

Miscellaneous

free

Display the amount of free and used memory (free -m to get info in MB)

htop

an improved top command

lsb_release

lsb_release -a prints version information for the Linux release you're running.

mkfifo

make FIFOs (named pipes)

pv, pipeview, pipebench

monitor the progress of data through a pipe

strace

trace system calls and signals

tee

read from standard input and write to standard output and files

uname

uname -a prints all system information, include machine name, kernel name & version.

watch

Execute a program periodically, showing output full screen

User Administration

groupadd / addgroup

Use groupadd, addgroup to create a new group. groupadd is the low-level utility. addgroup is the Debian more friendly version.

sudo groupadd groupname                      # Create a new group groupname
sudo groupadd -g 123 groupname               # ... and specify gid of the new group
sudo addgroup groupname
sudo addgroup --gid 123 groupname

useradd / adduser

Use useradd to create new user. useradd is the low-level utility. adduser is the Debian more friendly version.

sudo useradd username                        # Create a new user username
sudo useradd -s /bin/bash username           # ... specify the deafult shell in use for the user
sudo useradd -s /bin/bash -m username        # ... and create user home directory (from /etc/skel)
sudo useradd -u 1234 username                # Create a new user username with uid 1234
sudo useradd -g GRP1 -G GRP2,GRP3 username   # Specify primary and supplementary groups
sudo adduser username                        # Create a new user username
sudo adduser --system username               # Create a new system user (in the system uid range)
sudo adduser --shell /bin/bash username      # Use /bin/bash as default login shell

userdel / groupdel / deluser

Use userdel, groupdel, deluser to remove an existing user /group. deluser is the Debian more friendly version

sudo userdel username                        # Remove existing user username
sudo groupdel groupname                      # Remove existing group groupname
sudo deluser username                        # Remove existing user username
sudo deluser --group groupname               # Remove existing group groupname

usermod

Use usermod to modify an existing user

sudo usermod -g GRP1 username                # Modify the primary group of user username
sudo usermod -g GRP1 -G GRP2,GRP3 username   # ... and also the supplementary groups

X

xbindkeys / xbindkeys-config

• xbindkeys is a program that allows you to launch shell commands with your keyboard or your mouse under the X Window System. It links commands to keys or mouse buttons, using a configuration file. It's independent of the window manager and can capture all keyboard keys (ex: Power, Wake...).
• xbindkeys-config is an easy to use gtk program for configuring Xbindkeys (see [5].

XSel / XSelection

XSel is a command-line program for getting and setting the contents of the X selection and context-menu clipboard. Normally the former is only accessible by manually highlighting information and pasting it with the middle mouse button, and the latter is only accessible by using context-menu cut/copy/paste command.

xselection is an equivalent package on OpenSUSE.

On Cygwin, one has to use getclip, putclip or device /dev/clipboard.

xsel -p                                 # To get content of PRIMARY buffer (X selection)
xselection PRIMARY                      # (equivalent command for xselection)
xsel -b                                 # To get content of CLIPBOARD (context-menu clipboard)
xselection CLIPBOARD                    # (equivalent command for xselection)
echo sometext | xsel -p                 # To set PRIMARY buffer
echo sometext | xselection PRIMARY -    # (equivalent command for xselection)
xselection PRIMARY sometext             # (equivalent command for xselection)
echo sometext | xsel -b                 # To set CLIPBOARD buffer
echo sometext | xselection CLIPBOARD -  # (equivalent command for xselection)
xselection CLIPBOARD sometext           # (equivalent command for xselection)

xvkbd

xvkbd is a virtual (graphical) keyboard program for X Window System which provides facility to enter characters onto other clients (softwares) by clicking on a keyboard displayed on the screen. It can also be used to send keyboard events through command-line:

xvkbd -xsendevent -text text-string

Miscellaneous

xclip

xclip is a command line interface to the X11 clipboard. It can also be used for copying files, as an alternative to sftp/scp, thus avoiding password prompts when X11 forwarding has already been setup. Check this guide.

Miscellaneous

cksum / jacksum / md5sum / sha1sum / sum

cksum file                                              # Generate a CRC similar to CRC32, but with length appended.
cksum -o3 file                                          # Generate a CRC32 checksum (BSD only)
sum file                                                # Generate a checksum
md5sum file                                             # Hash file using MD5
sha1sum file                                            # Hash file using SHA1
jacksum -a crc32 file                                   # Hash using CRC32

diff

Use diff to compare 2 files together and/or to generate patch files

$ cp project_dir project_dir-patched     
$ cd project_dir-patched
$ vi somefile                                             # We start modifying the copy
$ vi someotherfile
$ cd ..
$ diff -u project_dir project_dir-patched >project.patch  # We generate the patch file (universal format)

Patch can be applied with the command patch (see below). Be careful regarding file names / directory structures when generating patch files! Sometimes you'll need to edit the generate patch files to fix the directory structure.

dog

dog is better than cat. It writes the contents of each give file, URL, or the standard input to standard output.

dtach

dtach simple program that emulates the detach feature of screen. A session in dtach is a single instance in which a program is running under the control of dtach. The program is disassociated from the original terminal, and is thus protected from your original terminal being disconnected for some reason.

echo

echo -e "Some text\n...on 2 lines..."                    # Enable interpretation of backslash escapes (must be quoted!)

expect

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.

od

od - dump files in octal and other formats

od -xc file                                              # Output file in ascii and hex output (2-byte, ! little-endian)
od -t x1 -a --width=32 testvar.sh                        # Output 32-byte at once

mc (Midnight Commander)

mc - Midnight Commander, a powerful file manager

• mc will only save the option ini file if there is a directory ~/.mc in home directory:

mkdir ~/.mc

• When using mc within Gnome Terminal, the Terminal menu shortcut key must be disabled in order to use mc shortcut key for exit (F10).
• Frequently used keyboard shortcut

key	description
F10	Exit Midnight Commander
Ctrl-O	open sub-shell (with selected tab as current directory)
Ins	select highlighted file and move to next

mrxvt

A multi-tab version of rxvt (terminal-emulator).

patch

Use patch to apply a patch previously generated with the command diff. patch use is pretty much straightforward, one mainly has to pay attention to directory structure / blanks.

Starting from the end of our example in diff above:

$ cd project_dir                 # We cd into project directory to patch
$ patch -lNp1<../project.patch   # We apply the patch in place

• The option l is to ignore white spaces (very handy when copying patch from internet page for instance).
• The option p1 tells patch to ignore the first (i.e. one level of) directory in the file name given in the patch header.

The value to use for option p depends actually on the patch header:

• First example - same root directory for both to and from file

--- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2/jpg.c       2009-08-25 16:06:05.242378300 +0200

• second example - different root directory between to and from file

--- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2-patched/jpg.c       2009-08-25 16:06:05.242378300 +0200

$ patch -lNp0 <project.patch                        # For first example header above
$ cd project_dir; patch -lNp1 <../project.patch     # For second example header above

recode

recode /cl../cr <dos.txt >mac.txt
recode /cr.. <mac.txt >unix.txt
recode ../cl <unix.txt >dos.txt
recode charset1/surface1..charset2/surface2 <input.txt >output.txt
recode /QP.. <qp.txt >normal.txt                                    # To convert quoted-printable text

charset		surface
us	ASCII (7 bits)	/cr	Carriage return as end of line (Mac text)
l1	ISO Latin-1 (ISO-8859-1, Western Europe, 8 bits)	/cl	Carriage return line feed as end of line (DOS text)
EUCJP	EUC-JP for Japanese (Unix)	/	Line feed as end of line (Unix text)
SJIS	Shift-JIS for Japanese (Microsoft)	/d1	Human readable bytewise decimal dump
ISO2022JP	Mail encoding for Japanese (7 bits)	/x1	Human readable bytewise hexidecimal dump
u2	UCS-2 (Universal Character Set, 2 bytes)	/64	Base64 encoded text
u8	UTF-8 (Universal Transformation Format, 8 bits)	/QP	Quoted-Printable encoded text

screen

Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.

sed

Moved to page dedicated to Sed.

Miscellaneous

iconv

locale encoding conversions

konwert

fancy encoding conversions

mimencode

binary file conversion for the mail.

reformime

MIME E-mail reformatting tool

uuencode, uudecode

binary file conversion for Unix.

wc

Counts lines, words and character in a file

xxd

Make a hexdump or do the reverse

$ echo 202122 | xxd -r -ps -                 # To decode a hexdump string
$ echo ' !"' | xxd -u -ps -                  # to code as a hexdump string (uppercase)