Linux Commands
File System
dd
Convert and copy a file.
dd returns current copy status when sent an HUP signal. Type in another shell:
while :; sleep 5; do killall -s SIGHUP1 dd; done
This will force dd to update its status every 5sec.
find
See here for further examples on how to combine find and xargs.
Frequent pitfall! The semi-colon ; must be escaped in Bash !
find . -name "*.jpg" # find files/directories whose name matches specified pattern
find . -path "*/cur/*" # find files/directories whose path spec matches specified pattern
find . -type d -name "cur" # find only directories whose name matches specified pattern
find . -exec echo '{}' \; # semi-colon must be escaped!
find . -exec echo one '{}' \; -exec echo two '{}' \; # to execute several commands on one match
find . -name \*.jpg | xargs echo # much faster version using xargs
find . -name \*.jpg | xargs -n 2 echo # limit #args passed to echo to 2 max.
find . -name \*.jpg -print0 | xargs -0 ls # Use NULL separator if files contains space or special char !!!
grep
grep -Rsl PATTERN [FILE] # Recursive, no error output, only list filename
grep BASIC-REG-EXP-PATTERN [FILE] # Use classic regexp (like "dma\|DMA")
egrep EXT-REG-EXP-PATTERN [FILE] # Same as grep -E. Use extended regexp (like "dma|DMA")
fgrep FIXED-STRINGS-REG-EXP [FILE] # Same as grep -F. Pattern is a list of strings to match.
grep -n PATTERN [FILE] # Print matched line numbers.
grep -- "-s" [FILE] # Search for text "-s"
grep -e "-s" [FILE] # Search for text "-s" - alternative solution
grep -R -include=*.in PATTERN * # Search recursively through folders, limiting to files matching pattern "*.in"
grep -R PATTERN *.in # Idem, but matching pattern "*.in" also applies to folders.
grep -o PATTERN [FILE] # (--only-matching) Print only the matched (non-empty) parts of a matching line.
Miscellaneous
- stat
- display file or file system status
- tree
- List the contents of directories in a tree-like format.
Network
autossh
autossh - Automatically restart SSH sessions and tunnels.
autossh by default monitors the ssh connection through a dedicated port to see whether the current ssh connection must be restarted. However the simpler is to use the ssh config option ServerAliveCountMax and ServerAliveInterval so that ssh exits automatically when the connection is down. In that case, autossh will then restart ssh automatically without need of additional monitoring.
Add to your ~/.ssh/config:
Host * ServerAliveCountMax 3 # default value actually ServerAliveInterval 15 # ssh will exit after 3 x 15s = 45s
Example of uses:
$ autossh -M 0 -f -D 1080 noekeon # -M 0 tells autossh to only restart ssh upon ssh's exit
multitee
multitee sends multiple inputs to multiple outputs.
Check this page.
Original is here, or can also be found here on Debian.
Here a patch to build it on Cygwin. The patch ports calls to BSD signal.h API (struct sigvec, sigvec(), sigmask(), sigblock()...) to the POSIX API (struct sigaction, sigaction(), sigprocmask(), sigfillset()...):
$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin
Example of use:
$ multitee 0-1,4,5 4> foo 5> bar # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0 # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1 # same as 'socat - FD:3'
netcat
TCP-IP swiss army knife (equivalent of the telnet program. Check wikipedia:netcat. Also known as command nc).
netstat
Print network connections, routing tables, interface statistics, masqurade connections, and multicast memberships
netstat -utpn #Active ports, tcp, socket program PID, numeric
netstat -lutpn #Listen ports, tcp, socket program PID, numeric
netstat -autpn #All (active and listen), tcp, socket program PID, numeric
netstat -rn #Kernel route table, numeric
When listing sockets (default output), you'll get an output like:
% netstat -at Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 *:time *:* LISTEN tcp 0 0 localhost:mysql *:* LISTEN tcp 0 0 andLinux.local:43449 windows-host:x11 ESTABLISHED |
% netstat -atn Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:37 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN tcp 0 0 192.168.11.150:43449 192.168.11.1:6000 ESTABLISHED |
- Local Address
- * or 0.0.0.0 means that the process accepts connection from any interface.
- 127.0.0.1 means it only accepts connection on localhost loopback (and so only connection that originates from local PC as well).
- Any other IP address means that the process listen on the given port at the given IP address
/etc/init.d/nscd
This is not really a command, but the init service Name Service Cache Daemon.
sudo /etc/init.d/nscd restart # To restart daemon and flush DNS cache
socat
Command-line utility that establishes two bidirectional byte streams and transfers data between them ([1]). socat is the more powerful version of netcat. Check the homepage. And also this page on Yobi for examples on how to use socat to bypass a proxy.
socat can easily tunnel connections:
socat -ly 'TCP4-LISTEN:143,reuseaddr,fork' PROXY:ton.imap.server:143|TCP:134.27.168.36:8080
socat can also be used as SSH ProxyCommand:
ProxyCommand socat -ly - 'PROXY:%h:%p|TCP:proxy.server:8080'
# Using socat v2.0.0 beta
ProxyCommand socat -ly - 'PROXY:%h:%p,proxyauth=user:pass|SSL,verify=0|PROXY:my.server:443,proxyauth=user:pass|TCP:big.brother.proxy:8080'
socat can be easily used as a replacement of telnet:
socat tcp:<host>:<port> - # <port> can be a port number or service name (telnet,imap...)
Some useful command-line options:
socat -ly # Writes messages to syslog instead of stderr; severity as defined with option -d
socat -v # Writes the transferred data to their target stream but also to stderr (text mode)
Package Management
dpkg
Package manager for Debian / Ubuntu. See Package Management#Ubuntu / Debian
rpm
System
Sudo
See also Ubuntu's page at [linux reference script security sudo ubuntu].
ls | sudo tee /root/somefile # Instead of redirection sudo ls > /root/somefile
ls | sudo tee -a /root/somefile # ... same but append instead
sudo sh -c "ls > /root/somefile" # Yet another way to do redirection with sudo
sudo !! # Repeat last command with root privilege
sudo -i # Get a root *login* shell
sudo su - # ... another solution
sudo -s # Root shell but keeping current environment
sudo su # ... another solution
sudo -k # Reset sudo password timeout
Miscellaneous
- free
- Display the amount of free and used memory (free -m to get info in MB)
- htop
- an improved top command
- lsb_release
- lsb-release -a prints version information for the Linux release you're running.
- mkfifo
- make FIFOs (named pipes)
- pv, pipeview, pipebench
- monitor the progress of data through a pipe
- strace
- trace system calls and signals
- tee
- read from standard input and write to standard output and files
- uname
- uname -a prints all system information, include machine name, kernel name & version.
- watch
- Execute a program periodically, showing output full screen
User Administration
groupadd / addgroup
Use groupadd, addgroup to create a new group. groupadd is the low-level utility. addgroup is the Debian more friendly version.
sudo groupadd groupname # Create a new group groupname
sudo groupadd -g 123 groupname # ... and specify gid of the new group
sudo addgroup groupname
sudo addgroup --gid 123 groupname
useradd / adduser
Use useradd to create new user. useradd is the low-level utility. adduser is the Debian more friendly version.
sudo useradd username # Create a new user username
sudo useradd -s /bin/bash username # ... specify the deafult shell in use for the user
sudo useradd -s /bin/bash -m username # ... and create user home directory (from /etc/skel)
sudo useradd -u 1234 username # Create a new user username with uid 1234
sudo useradd -g GRP1 -G GRP2,GRP3 username # Specify primary and supplementary groups
sudo adduser username # Create a new user username
sudo adduser --system username # Create a new system user (in the system uid range)
userdel / groupdel / deluser
Use userdel, groupdel, deluser to remove an existing user /group. deluser is the Debian more friendly version
sudo userdel username # Remove existing user username
sudo groupdel groupname # Remove existing group groupname
sudo deluser username # Remove existing user username
sudo deluser --group groupname # Remove existing group groupname
usermod
Use usermod to modify an existing user
sudo usermod -g GRP1 username # Modify the primary group of user username
sudo usermod -g GRP1 -G GRP2,GRP3 username # ... and also the supplementary groups
X
XSel / XSelection
XSel is a command-line program for getting and setting the contents of the X selection and context-menu clipboard. Normally the former is only accessible by manually highlighting information and pasting it with the middle mouse button, and the latter is only accessible by using context-menu cut/copy/paste command.
xselection is an equivalent package on OpenSUSE.
On Cygwin, one has to use getclip, putclip or device /dev/clipboard.
xsel -p # To get content of PRIMARY buffer (X selection)
xselection PRIMARY # (equivalent command for xselection)
xsel -b # To get content of CLIPBOARD (context-menu clipboard)
xselection CLIPBOARD # (equivalent command for xselection)
echo sometext | xsel -p # To set PRIMARY buffer
echo sometext | xselection PRIMARY - # (equivalent command for xselection)
xselection PRIMARY sometext # (equivalent command for xselection)
echo sometext | xsel -b # To set CLIPBOARD buffer
echo sometext | xselection CLIPBOARD - # (equivalent command for xselection)
xselection CLIPBOARD sometext # (equivalent command for xselection)
Miscellaneous
- xclip
- xclip is a command line interface to the X11 clipboard. It can also be used for copying files, as an alternative to sftp/scp, thus avoiding password prompts when X11 forwarding has already been setup. Check this guide.
Miscellaneous
diff
Use diff to compare 2 files together and/or to generate patch files
$ cp project_dir project_dir-patched
$ cd project_dir-patched
$ vi somefile # We start modifying the copy
$ vi someotherfile
$ cd ..
$ diff -u project_dir project_dir-patched >project.patch # We generate the patch file (universal format)
Patch can be applied with the command patch (see below). Be careful regarding file names / directory structures when generating patch files! Sometimes you'll need to edit the generate patch files to fix the directory structure.
dog
dog is better than cat. It writes the contents of each give file, URL, or the standard input to standard output.
echo
echo -e "Some text\n...on 2 lines..." # Enable interpretation of backslash escapes (must be quoted!)
patch
Use patch to apply a patch previously generated with the command diff. patch use is pretty much straightforward, one mainly has to pay attention to directory structure / blanks.
Starting from the end of our example in diff above:
$ cd project_dir # We cd into project directory to patch
$ patch -lNp1<../project.patch # We apply the patch in place
- The option l is to ignore white spaces (very handy when copying patch from internet page for instance).
- The option p1 tells patch to ignore the first (i.e. one level of) directory in the file name given in the patch header.
The value to use for option p depends actually on the patch header:
- First example - same root directory for both to and from file
--- outguess-0.2/jpg.c 2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2/jpg.c 2009-08-25 16:06:05.242378300 +0200
- second example - different root directory between to and from file
--- outguess-0.2/jpg.c 2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2-patched/jpg.c 2009-08-25 16:06:05.242378300 +0200
$ patch -lNp0 <project.patch # For first example header above
$ cd project_dir; patch -lNp1 <../project.patch # For second example header above
recode
recode /cl../cr <dos.txt >mac.txt
recode /cr.. <mac.txt >unix.txt
recode ../cl <unix.txt >dos.txt
recode charset1/surface1..charset2/surface2 <input.txt >output.txt
recode /QP.. <qp.txt >normal.txt # To convert quoted-printable text
| charset | surface | ||
|---|---|---|---|
| us | ASCII (7 bits) | /cr | Carriage return as end of line (Mac text) |
| l1 | ISO Latin-1 (ISO-8859-1, Western Europe, 8 bits) | /cl | Carriage return line feed as end of line (DOS text) |
| EUCJP | EUC-JP for Japanese (Unix) | / | Line feed as end of line (Unix text) |
| SJIS | Shift-JIS for Japanese (Microsoft) | /d1 | Human readable bytewise decimal dump |
| ISO2022JP | Mail encoding for Japanese (7 bits) | /x1 | Human readable bytewise hexidecimal dump |
| u2 | UCS-2 (Universal Character Set, 2 bytes) | /64 | Base64 encoded text |
| u8 | UTF-8 (Universal Transformation Format, 8 bits) | /QP | Quoted-Printable encoded text |
sed
Moved to page dedicated to Sed.
Miscellaneous
- iconv
- locale encoding conversions
- konwert
- fancy encoding conversions
- mimencode
- binary file conversion for the mail.
- reformime
- MIME E-mail reformatting tool
- uuencode, uudecode
- binary file conversion for Unix.
- wc
- Counts lines, words and character in a file
- xxd
- Make a hexdump or do the reverse
$ echo 202122 | xxd -r -ps - # To decode a hexdump string
$ echo ' !"' | xxd -u -ps - # to code as a hexdump string (uppercase)