Linux Commands

File System

dd

Convert and copy a file.

dd returns current copy status when sent an HUP signal. Type in another shell:

while :; sleep 5; do killall -s SIGHUP1 dd; done

This will force dd to update its status every 5sec.

find

See here for further examples on how to combine find and xargs.

Frequent pitfall! The semi-colon ; must be escaped in Bash !

find . -name "*.jpg"                                     # find files/directories whose name matches specified pattern
find . -path "*/cur/*"                                   # find files/directories whose path spec matches specified pattern
find . -type d -name "cur"                               # find only directories whose name matches specified pattern
find . -exec echo '{}' \;                                # semi-colon must be escaped!
find . -exec echo one '{}' \; -exec echo two '{}' \;     # to execute several commands on one match
find . -name \*.jpg | xargs echo                         # much faster version using xargs
find . -name \*.jpg | xargs -n 2 echo                    # limit #args passed to echo to 2 max.
find . -name \*.jpg -print0 | xargs -0 ls                # Use NULL separator if files contains space or special char !!!
find . -name 'dirname' -prune -o -iname '*.txt' -print   # Find all text files, but exclude any directory named dirname
find . -path './path' -prune -false -o -iname '*.txt' -print
                                                         # Find all text files, but exclude path ./path
find . -iname '*.zip' -exec sh -c 'unzip -l "$0" | egrep -o " [0-9]+ file"' '{}' \;
                                                         # How to use pipe in -exec.

grep

grep -Rsl PATTERN [FILE]            # Recursive, no error output, only list filename
grep BASIC-REG-EXP-PATTERN [FILE]   # Use classic regexp (like "dma\|DMA")
egrep EXT-REG-EXP-PATTERN [FILE]    # Same as grep -E. Use extended regexp (like "dma|DMA")
fgrep FIXED-STRINGS-REG-EXP [FILE]  # Same as grep -F. Pattern is a list of strings to match.
grep -n PATTERN [FILE]              # Print matched line numbers.
grep -- "-s" [FILE]                 # Search for text "-s"
grep -e "-s" [FILE]                 # Search for text "-s" - alternative solution
grep -R -include=*.in PATTERN *     # Search recursively through folders, limiting to files matching pattern "*.in"
grep -R PATTERN *.in                # Idem, but matching pattern "*.in" also applies to folders.
grep -o PATTERN [FILE]              # (--only-matching) Print only the matched (non-empty) parts of a matching line.

Miscellaneous

stat

display file or file system status

tree

List the contents of directories in a tree-like format.

Network

autossh

autossh - Automatically restart SSH sessions and tunnels.

autossh by default monitors the ssh connection through a dedicated port to see whether the current ssh connection must be restarted. However the simpler is to use the ssh config option ServerAliveCountMax and ServerAliveInterval so that ssh exits automatically when the connection is down. In that case, autossh will then restart ssh automatically without need of additional monitoring.

Add to your ~/.ssh/config:

Host *
  ServerAliveCountMax       3    # default value actually
  ServerAliveInterval       15   # ssh will exit after 3 x 15s = 45s

Example of uses:

$ autossh -M 0 -f -D 1080 noekeon       # -M 0 tells autossh to only restart ssh upon ssh's exit

multitee

multitee sends multiple inputs to multiple outputs.
Check this page.
Original is here, or can also be found here on Debian.

Here a patch to build it on Cygwin. The patch ports calls to BSD signal.h API (struct sigvec, sigvec(), sigmask(), sigblock()...) to the POSIX API (struct sigaction, sigaction(), sigprocmask(), sigfillset()...):

$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin

Example of use:

$ multitee 0-1,4,5 4> foo 5> bar             # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7                 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0   # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1                           # same as 'socat - FD:3'

netcat

TCP-IP swiss army knife (equivalent of the telnet program. Check wikipedia:netcat. Also known as command nc).

netstat

Print network connections, routing tables, interface statistics, masqurade connections, and multicast memberships

netstat -utpn      #Active ports, tcp, socket program PID, numeric
netstat -lutpn     #Listen ports, tcp, socket program PID, numeric
netstat -autpn     #All (active and listen), tcp, socket program PID, numeric
netstat -rn        #Kernel route table, numeric

When listing sockets (default output), you'll get an output like:

% netstat -at Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 *:time *:* LISTEN tcp 0 0 localhost:mysql *:* LISTEN tcp 0 0 andLinux.local:43449 windows-host:x11 ESTABLISHED

% netstat -atn Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:37 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN tcp 0 0 192.168.11.150:43449 192.168.11.1:6000 ESTABLISHED

Local Address

* or 0.0.0.0 means that the process accepts connection from any interface.

127.0.0.1 means it only accepts connection on localhost loopback (and so only connection that originates from local PC as well).

Any other IP address means that the process listen on the given port at the given IP address

/etc/init.d/nscd

This is not really a command, but the init service Name Service Cache Daemon.

sudo /etc/init.d/nscd restart                # To restart daemon and flush DNS cache

socat

Command-line utility that establishes two bidirectional byte streams and transfers data between them ([1]). socat is the more powerful version of netcat. Check the homepage. And also this page on Yobi for examples on how to use socat to bypass a proxy.

socat can easily tunnel connections:

socat -ly 'TCP4-LISTEN:143,reuseaddr,fork' PROXY:ton.imap.server:143|TCP:134.27.168.36:8080

socat can also be used as SSH ProxyCommand:

ProxyCommand socat -ly - 'PROXY:%h:%p|TCP:proxy.server:8080'
# Using socat v2.0.0 beta
ProxyCommand socat -ly - 'PROXY:%h:%p,proxyauth=user:pass|SSL,verify=0|PROXY:my.server:443,proxyauth=user:pass|TCP:big.brother.proxy:8080'

socat can be easily used as a replacement of telnet:

socat tcp:<host>:<port> -       # <port> can be a port number or service name (telnet,imap...)

Some useful command-line options:

socat -ly                       # Writes messages to syslog instead of stderr; severity as defined with option -d
socat -v                        # Writes the transferred data to their target stream but also to stderr (text mode)

Package Management

dpkg

Package manager for Debian / Ubuntu. See Package Management#Ubuntu / Debian

rpm

See Package Management#RPM's

System

Sudo

See also Ubuntu's page at [linux reference script security sudo ubuntu].

ls | sudo tee /root/somefile                # Instead of redirection sudo ls > /root/somefile
ls | sudo tee -a /root/somefile             # ... same but append instead
sudo sh -c "ls > /root/somefile"            # Yet another way to do redirection with sudo
sudo !!                                     # Repeat last command with root privilege
sudo -i                                     # Get a root *login* shell
sudo su -                                   # ... another solution
sudo -s                                     # Root shell but keeping current environment
sudo su                                     # ... another solution
sudo -k                                     # Reset sudo password timeout

Miscellaneous

free

Display the amount of free and used memory (free -m to get info in MB)

htop

an improved top command

lsb_release

lsb-release -a prints version information for the Linux release you're running.

mkfifo

make FIFOs (named pipes)

pv, pipeview, pipebench

monitor the progress of data through a pipe

strace

trace system calls and signals

tee

read from standard input and write to standard output and files

uname

uname -a prints all system information, include machine name, kernel name & version.

watch

Execute a program periodically, showing output full screen

User Administration

groupadd / addgroup

Use groupadd, addgroup to create a new group. groupadd is the low-level utility. addgroup is the Debian more friendly version.

sudo groupadd groupname                      # Create a new group groupname
sudo groupadd -g 123 groupname               # ... and specify gid of the new group
sudo addgroup groupname
sudo addgroup --gid 123 groupname

useradd / adduser

Use useradd to create new user. useradd is the low-level utility. adduser is the Debian more friendly version.

sudo useradd username                        # Create a new user username
sudo useradd -s /bin/bash username           # ... specify the deafult shell in use for the user
sudo useradd -s /bin/bash -m username        # ... and create user home directory (from /etc/skel)
sudo useradd -u 1234 username                # Create a new user username with uid 1234
sudo useradd -g GRP1 -G GRP2,GRP3 username   # Specify primary and supplementary groups
sudo adduser username                        # Create a new user username
sudo adduser --system username               # Create a new system user (in the system uid range)

userdel / groupdel / deluser

Use userdel, groupdel, deluser to remove an existing user /group. deluser is the Debian more friendly version

sudo userdel username                        # Remove existing user username
sudo groupdel groupname                      # Remove existing group groupname
sudo deluser username                        # Remove existing user username
sudo deluser --group groupname               # Remove existing group groupname

usermod

Use usermod to modify an existing user

sudo usermod -g GRP1 username                # Modify the primary group of user username
sudo usermod -g GRP1 -G GRP2,GRP3 username   # ... and also the supplementary groups

X

XSel / XSelection

XSel is a command-line program for getting and setting the contents of the X selection and context-menu clipboard. Normally the former is only accessible by manually highlighting information and pasting it with the middle mouse button, and the latter is only accessible by using context-menu cut/copy/paste command.

xselection is an equivalent package on OpenSUSE.

On Cygwin, one has to use getclip, putclip or device /dev/clipboard.

xsel -p                                 # To get content of PRIMARY buffer (X selection)
xselection PRIMARY                      # (equivalent command for xselection)
xsel -b                                 # To get content of CLIPBOARD (context-menu clipboard)
xselection CLIPBOARD                    # (equivalent command for xselection)
echo sometext | xsel -p                 # To set PRIMARY buffer
echo sometext | xselection PRIMARY -    # (equivalent command for xselection)
xselection PRIMARY sometext             # (equivalent command for xselection)
echo sometext | xsel -b                 # To set CLIPBOARD buffer
echo sometext | xselection CLIPBOARD -  # (equivalent command for xselection)
xselection CLIPBOARD sometext           # (equivalent command for xselection)

Miscellaneous

xclip

xclip is a command line interface to the X11 clipboard. It can also be used for copying files, as an alternative to sftp/scp, thus avoiding password prompts when X11 forwarding has already been setup. Check this guide.

Miscellaneous

diff

Use diff to compare 2 files together and/or to generate patch files

$ cp project_dir project_dir-patched     
$ cd project_dir-patched
$ vi somefile                                             # We start modifying the copy
$ vi someotherfile
$ cd ..
$ diff -u project_dir project_dir-patched >project.patch  # We generate the patch file (universal format)

Patch can be applied with the command patch (see below). Be careful regarding file names / directory structures when generating patch files! Sometimes you'll need to edit the generate patch files to fix the directory structure.

dog

dog is better than cat. It writes the contents of each give file, URL, or the standard input to standard output.

echo

echo -e "Some text\n...on 2 lines..."                    # Enable interpretation of backslash escapes (must be quoted!)

expect

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.

od

od - dump files in octal and other formats

od -xc file                                              # Output file in ascii and hex output (2-byte, ! little-endian)
od -t x1 -a --width=32 testvar.sh                        # Output 32-byte at once

patch

Use patch to apply a patch previously generated with the command diff. patch use is pretty much straightforward, one mainly has to pay attention to directory structure / blanks.

Starting from the end of our example in diff above:

$ cd project_dir                 # We cd into project directory to patch
$ patch -lNp1<../project.patch   # We apply the patch in place

• The option l is to ignore white spaces (very handy when copying patch from internet page for instance).
• The option p1 tells patch to ignore the first (i.e. one level of) directory in the file name given in the patch header.

The value to use for option p depends actually on the patch header:

• First example - same root directory for both to and from file

--- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2/jpg.c       2009-08-25 16:06:05.242378300 +0200

• second example - different root directory between to and from file

--- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2-patched/jpg.c       2009-08-25 16:06:05.242378300 +0200

$ patch -lNp0 <project.patch                        # For first example header above
$ cd project_dir; patch -lNp1 <../project.patch     # For second example header above

recode

recode /cl../cr <dos.txt >mac.txt
recode /cr.. <mac.txt >unix.txt
recode ../cl <unix.txt >dos.txt
recode charset1/surface1..charset2/surface2 <input.txt >output.txt
recode /QP.. <qp.txt >normal.txt                                    # To convert quoted-printable text

charset		surface
us	ASCII (7 bits)	/cr	Carriage return as end of line (Mac text)
l1	ISO Latin-1 (ISO-8859-1, Western Europe, 8 bits)	/cl	Carriage return line feed as end of line (DOS text)
EUCJP	EUC-JP for Japanese (Unix)	/	Line feed as end of line (Unix text)
SJIS	Shift-JIS for Japanese (Microsoft)	/d1	Human readable bytewise decimal dump
ISO2022JP	Mail encoding for Japanese (7 bits)	/x1	Human readable bytewise hexidecimal dump
u2	UCS-2 (Universal Character Set, 2 bytes)	/64	Base64 encoded text
u8	UTF-8 (Universal Transformation Format, 8 bits)	/QP	Quoted-Printable encoded text

sed

Moved to page dedicated to Sed.

Miscellaneous

iconv

locale encoding conversions

konwert

fancy encoding conversions

mimencode

binary file conversion for the mail.

reformime

MIME E-mail reformatting tool

uuencode, uudecode

binary file conversion for Unix.

wc

Counts lines, words and character in a file

xxd

Make a hexdump or do the reverse

$ echo 202122 | xxd -r -ps -                 # To decode a hexdump string
$ echo ' !"' | xxd -u -ps -                  # to code as a hexdump string (uppercase)