Linux Commands: Difference between revisions

From miki
Jump to navigation Jump to search
Line 922: Line 922:
sort file -o file # sort in-place
sort file -o file # sort in-place
sort file | sponge file # soak sorted file and write it back
sort file | sponge file # soak sorted file and write it back
</source>

=== sponge ===
'''sponge''' soaks up standard input and write to a file

<source lang=bash>
sed '...' file | grep '...' | sponge file # pipeline from and to same file!
</source>
</source>



Revision as of 17:30, 13 November 2013

File System

ack

ack is a grep-like program specifically for large source trees. It is distributed via package ack-grep. Add a symlink for convenience:

sudo ln -s /usr/bin/ack-grep /usr/local/bin/ack

Features:

  • It's blazingly fast because it only searches the stuff you want searched.
  • ack ignores most of the crap you don't want to search
  • Lets you specify file types to search, as in --perl or --nohtml.
  • Color highlighting of search results.
  • Uses real Perl regular expressions, not a GNU subset.
# grep pattern $(find . -type f | grep -v '\.svn')
ack pattern                                           # Ack ignore CVS, SVN dir 
# grep pattern $(find . -name '*.pl' -or -name '*.pm' -or -name '*.pod' | grep -v .svn)
ack --perl pattern                                    # Specify file type to search
ack -f --perl > all-perl-files                        # Create rapidly list of files

Supported file types (more here):

    --[no]asm          .asm .s
    --[no]batch        .bat .cmd
    --[no]binary       Binary files, as defined by Perl's -B op (default: off)
    --[no]cc           .c .h .xs
    --[no]cpp          .cpp .cc .cxx .m .hpp .hh .h .hxx
    --[no]css          .css
    --[no]hh           .h
    --[no]html         .htm .html .shtml .xhtml
    --[no]java         .java .properties
    --[no]js           .js
    --[no]make         Makefiles (including *.mk and *.mak)
    --[no]perl         .pl .pm .pod .t
    --[no]php          .php .phpt .php3 .php4 .php5 .phtml
    --[no]python       .py
    --[no]ruby         .rb .rhtml .rjs .rxml .erb .rake .spec
    --[no]shell        .sh .bash .csh .tcsh .ksh .zsh
    --[no]skipped      Files, but not directories, normally skipped by ack (default: off)
    --[no]text         Text files, as defined by Perl's -T op (default: off)
    --[no]vim          .vim
    --[no]xml          .xml .dtd .xsl .xslt .ent

dd

Convert and copy a file.

dd returns current copy status when sent an HUP signal. Type in another shell:

while :; sleep 5; do killall -s SIGHUP1 dd; done

This will force dd to update its status every 5sec.

find

See here for further examples on how to combine find and xargs.

Frequent pitfall! The semi-colon ; must be escaped in Bash !

find . -name "*.jpg"                                     # find files/directories whose name matches specified pattern
find . -path "*/cur/*"                                   # find files/directories whose path spec matches specified pattern
find . -type d -name "cur"                               # find only directories whose name matches specified pattern
find . -exec echo '{}' \;                                # semi-colon must be escaped!
find . -exec echo one '{}' \; -exec echo two '{}' \;     # to execute several commands on one match
find . -name \*.jpg | xargs echo                         # much faster version using xargs
find . -name \*.jpg | xargs -n 2 echo                    # limit #args passed to echo to 2 max.
find . -name \*.jpg -print0 | xargs -0 ls                # Use NULL separator if files contains space or special char !!!
find . -name 'dirname' -prune        -o -iname '*.txt' -print   
                                                         # Find all text files, but exclude any directory named dirname
find . -name 'dirname' -prune -false -o -iname '*.txt'   # ... idem (using -false, but no longer -print)
find . -path './path'  -prune -false -o -iname '*.txt'   # Find all text files, but exclude path ./path
find . -iname '*.zip' -exec sh -c 'unzip -l "$0" | egrep -o " [0-9]+ file"' '{}' \;
                                                         # How to use pipe in -exec.
find www -name '*.pdf' -printf %P\\n                     # Print PDF files in www/ but remove the leading www/ in output (%P)

Accelerating find:

find / -mount -name "*splash*"                           # Only search on CURRENT file system (same as find -xdev)

With mv, use -t DIRECTORY to give destination directory first:

find . -size +100M -print0 | xargs -0 mv -t bigfiles/    # -t DIR inverts usual SOURCE... DESTINATION order
find . -size +100M -exec mv -t bigfiles/ {} +            # ... idem, without xargs

Equivalence between -exec command \; and xargs -i, and -exec command {} + and xargs:

find . -name "*.txt" -exec echo '{}' is found. \;                # ONE at a time, but {} can be anywhere - and MULTIPLE times
find . -name "*.txt" -print0 | xargs -0 -i echo '{}' is found.   # ... same, with xargs
find . -name "*.txt" -exec echo The files are '{}' +             # FASTER - much like xargs. Only append AT THE END!
find . -name "*.txt" -print0 | xargs -0 echo The files are       # ... same with xargs

FSlint

FSlint is a utility to find and clean various forms of lint on a filesystem. I.E. unwanted or problematic cruft in your files or file names. For example, one form of lint it finds is duplicate files.

getfattr / setfattr

getfattr and setfattr gets/sets extended attributes of filesystem objects.

!!! These tools are really misleading, and many attributes are hidden. Usually you need to list the attributes explicitly !!!

getfattr -d file                      # This SUPPOSEDLY returns *ALL* attributes, but actually *MANY* missing
getfattr -m - file                    # This MIGHT return more attributes
getfattr -n system.ntfs_attrib file   # View ntfs-3g special attributes
getfattr -d -e hex file               # View attribute values in HEX

grep / zgrep

grep -Rsl PATTERN [FILE]            # Recursive, no error output, only list filename
grep BASIC-REG-EXP-PATTERN [FILE]   # Use classic regexp (like "dma\|DMA")
egrep EXT-REG-EXP-PATTERN [FILE]    # Same as grep -E. Use extended regexp (like "dma|DMA")
fgrep FIXED-STRINGS-REG-EXP [FILE]  # Same as grep -F. Pattern is a list of strings to match.
grep -n PATTERN [FILE]              # Print matched line numbers.
grep -- "-s" [FILE]                 # Search for text "-s"
grep -e "-s" [FILE]                 # Search for text "-s" - alternative solution
grep -R -include=*.in PATTERN *     # Search recursively through folders, limiting to files matching pattern "*.in"
grep -R PATTERN *.in                # Idem, but matching pattern "*.in" also applies to folders.
grep -o PATTERN [FILE]              # (--only-matching) Print only the matched (non-empty) parts of a matching line.

zgrep works like grep, but also accepts compressed (.gz) files.

zgrep PATTERN *.txt *.txt.gz        # Search in .txt file (compressed or not)

gzip / gunzip / zcat

gzip -c FILE                        # Compress FILE to stdout
gunzip -c FILE                      # Uncompress FiLe to stdout
zcat FILE                           # Equivalent to gunzip -c FILE

ls

Listing directory content:

ls --full-time               # To list files full date & time
ls -d directory-name         # List directory entries, not content
                             #  eg. ls -d /etc/rc*
ls -lS | head                # List biggest files first - print only first 10 entries
ls -lt | head -n 20          # List most recent files first, print only 20 first entries

ntfsundelete

ntfsundelete recover a delete file from an NTFS volume. Does not scan the disk.

See [1], [2].

scalpel

scalpel recover files using a header / footer database (see [3]).

Install scalpel, then edit /etc/scalpel/scalpel.conf to select which file types to recover (based on header/footer).

sudo apt-get install scalpel
sudo vi /etc/scalpel/scalpel.conf

Here we enable jpg and mp3 (see [4] for more):

# GIF and JPG files (very common)
#	gif	y	5000000		\x47\x49\x46\x38\x37\x61	\x00\x3b
#  	gif	y 	5000000		\x47\x49\x46\x38\x39\x61	\x00\x00\x3b
 	jpg	y	200000000	\xff\xd8\xff\xe0\x00\x10	\xff\xd9
        jpg     y       200000000       \xff\xd8\xff\xe1                \xff\xd9 
# MP3
       mp3     y       8000000 \xFF\xFB??\x44\x00\x00
       mp3     y       8000000 \x57\x41\x56\45            \x00\x00\xFF\
       mp3     y       8000000 \xFF\xFB\xD0\            \xD1\x35\x51\xCC\
       mp3     y       8000000 \x49\x44\x33\
       mp3     y       8000000 \x4C\x41\x4D\x45\

Start scalpel:

sudo scalpel /dev/sda1 -o output_dir      # output_dir must *not* be on disk being scanned


tar

tar archives and restores timestamps. When run as root, tar will also restore ownership and permissions.

tar -cvzf archive.tgz somedir/                             # Archive & GZIP compress the directory somedir/ (recursively)
tar -cvjf archive.tar.bz2 somedir/                         # ... Use BZIP2 insteadl
tar -cvzf archive.tgz --exclude=notme somedir/             #  ... exclude any file matching name
tar -cvzf archive.tgz --exclude=somedir/not/me somedir/    #  ... exclude file at specific location

tar -xvzf archive.tgz                                      # Extract archive (will create somedir/)
tar -xvzf archive.tgz -C targetdir                         # ... to specified directory
tar -tvzf archive.tgz                                      # List the content (equiv. of tar --list -vzf ...)

testdisk / photorec

testdisk scans and repairs disk partitions. photorec recovers lost files from harddisk, digital camera and cdrom (including audio and photo files).

photorec /log /d output_dir /dev/sda1

Miscellaneous

badblocks
An utility to detect bad sectors on USB flash drive or so.
dosfslabel
Change FAT16/FAT32 disk label
sudo dosfslabel /dev/sdc1 UBUNTULIVE
install-mbr
Install MS-like Master Boot Record (see [5])
sudo apt-get install mbr
install-mbr -i n -p D -t 0 /dev/sda        # Fix MBR on /dev/sda
                                           # -i interrupt n = none (do not display a MBR prompt)
                                           # -p partition D = boot the partition with the bootable flag set
                                           # -t timeout 0 = do not wait to boot
dd if=/dev/sda of=opensource.mbr bs=512 count=1
install-mbr -i n -p D -t 0 opensource.mbr  # Create a file opensource.mbr containing the generated MBR
mkfs
Format disk utility
sudo mkfs.vfat -F 32 -c /dev/sdc1
ms-sys
Install MS-compatible Master Boot Record (similar to FIXMBR on WinXP rescue console, see [6])
# Get ms-sys from http://ms-sys.sourceforge.net/#Download
sudo apt-get install gettext
make
sudo make install
sudo ms-sys -m /dev/sdb
relocntfs
relocntfs (see [7]) is more or less the equivalent of the FIXBOOT command in the Windows Recovery Console. It is among other available on the Trinity Rescue Kit. For instance, to fix the boot record of a windows partition at /dev/sda2 issue the commands:
relocntfs /dev/sda2        
relocntfs -w /dev/sda2                    # To actually write the new boot record
stat
display file or file system status (like file access time, creation time, modification time...)
tree
List the contents of directories in a tree-like format.

Network

autossh

autossh - Automatically restart SSH sessions and tunnels.

autossh by default monitors the ssh connection through a dedicated port to see whether the current ssh connection must be restarted. However the simpler is to use the ssh config option ServerAliveCountMax and ServerAliveInterval so that ssh exits automatically when the connection is down. In that case, autossh will then restart ssh automatically without need of additional monitoring.

Add to your ~/.ssh/config:

Host *
  ServerAliveCountMax       3    # default value actually
  ServerAliveInterval       15   # ssh will exit after 3 x 15s = 45s

Example of uses:

$ autossh -M 0 -f -D 1080 noekeon       # -M 0 tells autossh to only restart ssh upon ssh's exit

dig

dig stands for domain information groper. It replaces the deprecated nslookup. It comes with package dnsutils:

sudo apt-get install dnsutils

ip

ip show / manipulate routing, devices, policy routing and tunnels:

IPETH0=$(ip addr show eth0 | perl -lne 'print for /inet[^0-9]*([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)/')      # Get local ip address

multitee

multitee sends multiple inputs to multiple outputs.
Check this page.
Original is here, or can also be found here on Debian.

Here a patch to build it on Cygwin. The patch ports calls to BSD signal.h API (struct sigvec, sigvec(), sigmask(), sigblock()...) to the POSIX API (struct sigaction, sigaction(), sigprocmask(), sigfillset()...):

$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin

Example of use:

$ multitee 0-1,4,5 4> foo 5> bar             # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7                 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0   # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1                           # same as 'socat - FD:3'

netcat

TCP-IP swiss army knife (equivalent of the telnet program. Check wikipedia:netcat. Also known as command nc).

See also Using Netcat (Linux Tips).

netstat

Print network connections, routing tables, interface statistics, masqurade connections, and multicast memberships

netstat -utpn      #Active ports, tcp, socket program PID, numeric
netstat -lutpn     #Listen ports, tcp, socket program PID, numeric
netstat -autpn     #All (active and listen), tcp, socket program PID, numeric
netstat -rn        #Kernel route table, numeric

When listing sockets (default output), you'll get an output like:

% netstat -at

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 *:time                  *:*                     LISTEN
tcp        0      0 localhost:mysql         *:*                     LISTEN
tcp        0      0 andLinux.local:43449    windows-host:x11        ESTABLISHED
% netstat -atn

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:37              0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN
tcp        0      0 192.168.11.150:43449    192.168.11.1:6000       ESTABLISHED
Local Address
* or 0.0.0.0 means that the process accepts connection from any interface.
127.0.0.1 means it only accepts connection on localhost loopback (and so only connection that originates from local PC as well).
Any other IP address means that the process listen on the given port at the given IP address

nmap

nmap is the network exploration tool and security / port scanner.

nmap localhost                    # Scan and print all open ports on 'localhost'
nmap -sP hostname                 # Simply test whether 'hostname' is accessible

One can use nmap in a bash script to test whether a given host is accessible, but it turns out that ping is faster:

# Using nmap
nmap -sP $1 2>/dev/null | grep -q "1 host up"    # Too slow, does not always detect hosts

# ... or ping (faster)
if [ "$HOST_MACH" = "cygwin" ]; then
    ping $1 1 1 2>/dev/null | grep -q "1 packets received"
else
    ping -c 1 $1 2>/dev/null | grep -q "1 received"
fi

nm-tool

nm-tool is an utility to report NetworkManager state and devices.

nm-tool                   # Print IP addresses, DNS servers, etc.

/etc/init.d/nscd

This is not really a command, but the init service Name Service Cache Daemon.

sudo /etc/init.d/nscd restart                # To restart daemon and flush DNS cache

rpcinfo

rpcinfo reports RPC information

rpcinfo -p              # Probe the portmapper on host, and print a list of all registered RPC programs

socat

Command-line utility that establishes two bidirectional byte streams and transfers data between them ([8]). socat is the more powerful version of netcat. Check the homepage. And also this page on Yobi for examples on how to use socat to bypass a proxy.

# All these commands listen on localhost:53805, and forward to server:53806
socat TCP4-LISTEN:53805                TCP:server:53806      # Listen, and close as soon socket closes
socat TCP4-LISTEN:53805,reuseaddr,fork TCP:server:53806      # Idem, but fork so that listener continues after socket closes

socat -v ...                                                 # Send txfed data to stderr, with some formatting (text format)
socat -x ...                                                 # ... idem in hex format (can be combined with -v)
socat -ly                                                    # Writes messages to syslog instead of stderr; severity as defined with option -d

socat can easily tunnel connections:

socat -ly 'TCP4-LISTEN:143,reuseaddr,fork' PROXY:ton.imap.server:143|TCP:134.27.168.36:8080

socat can also be used as SSH ProxyCommand:

ProxyCommand socat -ly - 'PROXY:%h:%p|TCP:proxy.server:8080'
# Using socat v2.0.0 beta
ProxyCommand socat -ly - 'PROXY:%h:%p,proxyauth=user:pass|SSL,verify=0|PROXY:my.server:443,proxyauth=user:pass|TCP:big.brother.proxy:8080'

socat can be easily used as a replacement of telnet:

socat tcp:<host>:<port> -       # <port> can be a port number or service name (telnet,imap...)
QUESTION TO MR DOEGOX:
- WHY -s in SED?
- Where does it get info on % for socat?

$ socat echo -
$ socat echo STDIO
$ socat echo STDOUT%STDIN
$ socat echo STDIO%STDIO
$ socat echo -%-
$ socat echo FD:1%FD:0
$ socat echo 1%0

/usr/local/bin/socat TCP:127.0.0.1:1234 -%EXEC:"awk 'BEGIN{print \"BANNER\";fflush()}/BANNER/{next}//{print;fflush()}'"

Testing socat:
- /usr/local/bin/socat - TCP-LISTEN:1234
- /usr/local/bin/socat TCP:127.0.0.1:1234 EXEC:"/bin/sed -s 's/foo/bar/g'"%EXEC:"/bin/sed -u 's/toto/tata/g'"

  First open a port for listening and connect it to STDIO
  In another shell, connect and do the black magic...


reference:
- http://eros.ph0k.eu/~nizox/twin/V2/modules/socat-1.6.0.0/EXAMPLES
- http://www.rschulz.eu/2008/09/ssh-proxycommand-without-netcat.html
		$ exec 3<>/dev/tcp/zeus.yobi.be/22;(cat <&3 & );cat >&3
		SSH-2.0-OpenSSH_5.1p1 Debian-3
- http://thesmithfam.org/blog/2006/05/23/bash-socket-programming-with-devtcp-2/
- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464
- http://www.mtu.net/~engstrom/ssh-proxy.php



ATTEMPT 3:
Window 1: socat -v -%'EXEC:/home/beq06659/.ssh/mys2.sh' TCP-LISTEN:1234
Window 2: /usr/local/bin/socat -ly -v -%"EXEC:sed -unf /home/beq06659/.ssh/sedy" "PROXY:ftp.noekeon.org:22|TCP:localhost:1234"

File mys2.sh:
	#! /bin/bash
	echo -e "HTTP/1.1 200 Connection established\r\n\r\n"
	cat
File sedy:
	1 s/^/SSH-2.0-OpenSSH_5.1\n/p; /SSH-2.0-OpenSSH_5.1/d; /SSH-2.0-OpenSSH_5.1/! p

LESSONS LEARNED:
- READ THE CHANGES DOC, especially for BETA SW
- SOCKET = bidir --> a filter = 4 descriptors
- PIPE = unidir --> a filter = 2 descriptor (stdin/stdout)

Internet

httrack

httrack is a offline browser; it copies websites to a local directory.

# Example: checking out progit boot
# -d                       stay on same principal domain
# -w                       mirror web sites
# -O /var/www              path for mirror/logfiles+cache
# -I0                      don't make an index
# -%v                      display on screen filesnames downloaded
# -P 127.0.0.1:8118        Proxy settings
# '+progit.org/figures/*'  +filter
# '-progit.org/book/*/*'   -filter
httrack http://progit.org/book/ -d -w -O /var/www -I0 -%v '+progit.org/figures/*' '-progit.org/book/*/*' [-P 127.0.0.1:8118]
sed -ri \"s_'http://progit.org/book'_'index.html'_\" /var/www/progit.org/book/*.html

# Example 2
# -W                       mirror web sites, semi-automatic (ask question)
httrack http://www.cplusplus.com/ -W -O /var/www -%v '-www.cplusplus.com/forum/*' '-www.cplusplus.com/src/*' '-www.cplusplus.com/member/*' [-P 127.0.0.1:8118]

# Example 3 - Use different path for cache/log files
#  -O /var/www,/tmp/httrack   path for mirror,path cache and logfiles
httrack http://www.argospress.com/Resources/systems-engineering/index.html -d -O /var/www,/tmp/httrack -I0 -%v [-P 127.0.0.1:8118]
sed -ri '/wms.assoc-amazon.com/{N; d}' /var/www/www.argospress.com/Resources/systems-engineering/*.htm

# Example 4 - Limit depth
#  -r2                     set mirror depth to 2 (i.e. given address + 1st links)
httrack http://powerman.name/doc/asciidoc -r2 -w -I0 [-P 127.0.0.1:8118]

Serial

stty

stty changes and prints terminal line settings. It usually comes pre-installed.

stty -F /dev/ttyUSB0 -a         # Print all settings in human-readable form
stty -F /dev/ttyUSB0 --save     # Print all settings in stty-readable form

setserial

setserial get/set Linux serial port information.

See for instance [9] for usage.

Package Management

dpkg

Package manager for Debian / Ubuntu. See Package Management#Ubuntu / Debian

rpm

See Package Management#RPM's

System

dmidecode

dmidecode is a tool to decode DMI (SMBIOS) tables:

sudo dmidecode
...
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
	Vendor: Dell Inc.
	Version: A19
	Release Date: 12/21/2009
	Address: 0xF0000
	Runtime Size: 64 kB
	ROM Size: 1728 kB
	Characteristics:
		ISA is supported
		PCI is supported
		PC Card (PCMCIA) is supported
		PNP is supported
		BIOS is upgradeable
		BIOS shadowing is allowed
...

iwconfig

iwconfig configures or queries a wireless network interface:

iwconfig
# # iwconfig
# lo        no wireless extensions.
# 
# eth0      no wireless extensions.
# 
# wlan0     IEEE 802.11abg  ESSID:off/any  
#           Mode:Managed  Access Point: Not-Associated   Tx-Power=15 dBm   
#           Retry  long limit:7   RTS thr:off   Fragment thr:off
#           Encryption key:off
#           Power Management:off
iwconfig wlan0         # Limit to wlan0

lsmod / modinfo / modprobe

lsmod                              # Show the status of modules in the Linux Kernel
modinfo iwlagn                     # Show information about a Linux Kernel module
modprobe -r iwlagn                 # Remove module from the Linux Kernel
modprobe iwlagn 11n_disable=1      # Add module to the Linux Kernel, with specific parameters

loadkeys

loadkeys loads keyboard translation tables.

Use loadkeys to change the keyboard layout in console mode (i.e. Ctrl-Alt-F1 console).

sudo loadkeys be                   # Load Belgian layout

lspci

lspci                              # List all PCI devices
lspci -v                           # List all PCI devices (verbose)

newgrp

newgrp log in to a new group

 
newgrp - <group>       # Start a new session with <group> as primary group. - means to reinitialize environment.
umask 002              # Make sure all files / directories we create have group access

To run newgrp in a script, use the following construction [10]:

newgrp groupb << END
    touch "myscript_output.txt"
END

Or better yet use the sg command instead.

pidinfo

Find the process ID of a running program

pidof apache2                      # Find all pid of apache2

pgrep

Prints PID of the process matching given pattern. Never match pgrep itself!

ps -fp $(pgrep autossh)            # Much better that "ps -aef | grep autossh", which match grep process itself

pkill

Send a signal (default TERM) to process matching given pattern. Never match pkill itself!

pkill -0 autossh                   # Test if process "autossh" is running

ps

ps -ef                             # All processes            - full format

ps afl                             # Proc w ttys, all users   - long + forest
ps flx                             # Proc w/o ttys            - long + forest
ps aflx                            # All processes, all users - long + forest

ps au                              # Proc w ttys, all users   - user output
ps ux                              # Proc w/o ttys            - user output
ps aux                             # All processes, all users - user output

ps axs                             # All process, with signals

pv

pipe viewer — terminal-based tool for monitoring the progress of data through a pipeline

pv -cN source < linux-2.2.20.tar.bz2 | bzcat | pv -CN bzcat | gzip -9 | pv -cN gzip > linux-2.2.20.tar.gz
   source: 4.48MB 0:02:01 [9.84kB/s] [========>                        ] 29% ETA 0:04:44
    bzcat:   26MB 0:02:01 [ 192kB/s] [               <=>                               ]
     gzip: 5.58MB 0:02:01 [  64kB/s] [                         <=>                     ]

sg

sg executes command as different group ID.

umask 002                     # Ensure that all files written will be group accessible
sg - <group> -c "mkdir foo"   # Run commands as group <group>

The following construction restart the current script under a different group (from [11]):

if ! [ "${SBREADY:=false}" = true -o $(id -ng) = ${NEEDGRP:=wheel} ] ; then
    export SBREADY=true
    exec sg $NEEDGRP "$0" "$@"
fi

sudo / gksudo

See also Ubuntu's page at [linux reference script security sudo ubuntu].

ls | sudo tee /root/somefile                # Instead of redirection sudo ls > /root/somefile
ls | sudo tee -a /root/somefile             # ... same but append instead
sudo sh -c "ls > /root/somefile"            # Yet another way to do redirection with sudo
sudo !!                                     # Repeat last command with root privilege
sudo -i                                     # Get a root *login* shell
sudo su -                                   # ... another solution
sudo -s                                     # Root shell but keeping current environment
sudo su                                     # ... another solution
sudo -k                                     # Reset sudo password timeout

gksudo is the graphical equivalent of sudo:

gksudo gedit /etc/fstab

showkey

showkey examine the codes sent by the keyboard (linux kernel only - for X keyboard events, see xev!).

showkey -k                                  # Dump key codes (default)
showkey -s                                  # Dump scan codes

timeout / timelimit

timeout / timelimit are small binary to run a command with a given time limit. More exactly they send a given signal to the child process after some specified time. See also Perl - kill on ALARM signal).

timeout -15 30 mplayer dvd://              # Show 30s of DVD movie, then kill mplayer with a TERM signal

Memory related

free
Display the amount of free and used memory (free -m to get info in MB)
free
free -m       # Show in megabyte
/proc/meminfo
All memory usage information
cat /proc/meminfo
vmstat
Summary of memory usage
vmstat

Miscellaneous

htop
an improved top command
lsb_release
lsb_release -a prints version information for the Linux release you're running.
mkfifo
make FIFOs (named pipes)
pv, pipeview, pipebench
monitor the progress of data through a pipe
strace
trace system calls and signals
tee
read from standard input and write to standard output and files
uname
uname -a prints all system information, include machine name, kernel name & version.
watch
Execute a program periodically, showing output full screen

User / Group Administration

groupadd / addgroup

Use groupadd, addgroup to create a new group. groupadd is the low-level utility. addgroup is the Debian more friendly version.

sudo groupadd groupname                      # Create a new group groupname
sudo groupadd -g 123 groupname               # ... and specify gid of the new group
sudo addgroup groupname
sudo addgroup --gid 123 groupname

useradd / adduser

Use useradd to create new user. useradd is the low-level utility. adduser is the Debian more friendly version.

sudo useradd username                        # Create a new user username
sudo useradd -s /bin/bash username           # ... specify the deafult shell in use for the user
sudo useradd -s /bin/bash -m username        # ... and create user home directory (from /etc/skel)
sudo useradd -u 1234 username                # Create a new user username with uid 1234
sudo useradd -g GRP1 -G GRP2,GRP3 username   # Specify primary and supplementary groups
sudo adduser username                        # Create a new user username
sudo adduser --system username               # Create a new system user (in the system uid range)
sudo adduser --shell /bin/bash username      # Use /bin/bash as default login shell

userdel / groupdel / deluser

Use userdel, groupdel, deluser to remove an existing user /group. deluser is the Debian more friendly version

sudo userdel username                        # Remove existing user username
sudo groupdel groupname                      # Remove existing group groupname
sudo deluser username                        # Remove existing user username
sudo deluser --group groupname               # Remove existing group groupname

usermod

Use usermod to modify an existing user

sudo usermod -g GRP1 username                # Modify the primary group of user username
sudo usermod -g GRP1 -G GRP2,GRP3 username   # ... and also the supplementary groups

gpasswd

Use gpasswd to administer /etc/group and /etc/gshadow

sudo gpasswd -a $USER fuse                   # Add self to group 'fuse'

X

disper

TO install:

sudo add-apt-repository ppa:wvengen/ppa
sudo apt-get update
sudo apt-get install

Some examples of use:

disper -s               # Only enable primary screen (laptop screen)
disper -S               # Only enable secondary screen (external monitor)
disper -d auto -e       # Auto-detect + enable extended mode
disper -d auto -c       # clone mode

xbindkeys / xbindkeys-config

  • xbindkeys is a program that allows you to launch shell commands with your keyboard or your mouse under the X Window System. It links commands to keys or mouse buttons, using a configuration file. It's independent of the window manager and can capture all keyboard keys (ex: Power, Wake...).
  • xbindkeys-config is an easy to use gtk program for configuring Xbindkeys (see [12].

xclip

xclip is a command line interface to the X11 clipboard. It can also be used for copying files, as an alternative to sftp/scp, thus avoiding password prompts when X11 forwarding has already been setup. Check this guide.

xdpyinfo

Display information utility for X

xdpyinfo | grep dimensions               # Return dimension of the desktop

XSel / XSelection

XSel is a command-line program for getting and setting the contents of the X selection and context-menu clipboard. Normally the former is only accessible by manually highlighting information and pasting it with the middle mouse button, and the latter is only accessible by using context-menu cut/copy/paste command.

xselection is an equivalent package on OpenSUSE.

On Cygwin, one has to use getclip, putclip or device /dev/clipboard.

xsel -p                                 # To get content of PRIMARY buffer (X selection)
xselection PRIMARY                      # (equivalent command for xselection)
xsel -b                                 # To get content of CLIPBOARD (context-menu clipboard)
xselection CLIPBOARD                    # (equivalent command for xselection)
echo sometext | xsel -p                 # To set PRIMARY buffer
echo sometext | xselection PRIMARY -    # (equivalent command for xselection)
xselection PRIMARY sometext             # (equivalent command for xselection)
echo sometext | xsel -b                 # To set CLIPBOARD buffer
echo sometext | xselection CLIPBOARD -  # (equivalent command for xselection)
xselection CLIPBOARD sometext           # (equivalent command for xselection)

xvkbd

xvkbd is a virtual (graphical) keyboard program for X Window System which provides facility to enter characters onto other clients (softwares) by clicking on a keyboard displayed on the screen. It can also be used to send keyboard events through command-line:

xvkbd -xsendevent -text text-string

xwininfo

xwininfo is the window information utility for X. Could be used to find the geometry of a window. See here

xwininfo                               # Return information on the clicked windows
xwininfo -root                         # ... or on the root window
xwininfo -root |awk '/Width/{print $2} # Get Width of the current desktop

Text Manipulation

column

Format input into multiple columns

cat | column -l
123 456
98765 98765
^D
# 123   456
# 98765 98765

cut

TBC

paste

TBC

fold

fold -w 40 FILE              # Wrap FILE to 40-char width

expand

expand -t 4 FILE             # Convert tabs to spaces (4)
expand -t 4,10 FILE          # Set explicit tab positions

fmt

TBC

nl

nl numbers lines of files

nl -ba FILE                  # Number *all* lines of FILE

rl, shuf

rl is a command-line tool that reads lines from an input file or stdin, randomizes the lines and outputs a specified number of lines (requires package randomize-lines).

shuf generates random permutations.


They are alternative to sort -R (note that the latter does not work with duplicate lines).

sed

Moved to page dedicated to Sed.

sort

sort is the well-know text sort utility.

sort -R shuffle.txt                           # shuffle the lines (use random hash) - fail with duplicate lines
nl -ba file | sort -R | sed 's/^.*[0-9]\t//'  # ... idem, but workaround for dup lines

sort by default takes into account the current locale. To get the traditional sort order that uses native byte values:

LC_ALL=C sort myfile.txt                      # traditional sort, indep of current locale

Use -o file to sort in-place, or use the sponge utility:

sort file -o file                             # sort in-place
sort file | sponge file                       # soak sorted file and write it back

Development

addr2line

addr2Line converts addresses into file names and line numbers.

ar

ar creates, modifies, and extracts from archives. It can be used to generate static libraries (.a files) for use with gcc.

diff

Use diff to compare 2 files/directories together and/or to generate patch files. Use colordiff (package colordiff) to have colored output.!

diff    old new                        # View difference (default ed script format)
diff -u old new >new.patch             # Generate a patch file (universal format)
diff -y old new                        # View difference side-by-side (as with sdiff)

#Color with DIFF:
colordiff -u old new                   # colordiff can also colorize a diff output package colordiff
diff old new | colordiff               # ... idem

#Color with WDIFF:
wdiff -n old new | colordiff           # But need option '-n'

#Piping to LESS with COLORS
diff -u old new | colordiff | less -R  # To keep color with pager

#DIFF / WDIFF combo
diff --strip-trailing-cr -bu old new | wdiff -d | colordiff | less -R             # Handy when wdiff miss some options...

Patch can be applied with the command patch (see below). Be careful regarding file names / directory structures when generating patch files! Sometimes you'll need to edit the generated patch files to fix the directory structure.

patch

Use patch to apply a patch previously generated with the command diff. patch use is pretty much straightforward, one mainly has to pay attention to directory structure / blanks.

Complete example using diff:

$ cp project project-patched     
$ cd project-patched
$ vi somefile                                             # We start modifying the copy
$ cd ..
$ diff -u project project-patched >project.patch          # We generate the patch file (universal format)
$ cd project                                              # We cd into project directory to patch
$ patch -lNp1<../project.patch                            # We apply the patch in place
  • -l — ignore white spaces (very handy when copying patch from internet page for instance).
  • -N — ignore patch that seems reversed or already applied (same as --forward).
  • -p1 — tells patch to ignore the first (i.e. one level of) directory in the file name given in the patch header.

The value to use for option p depends actually on the patch header:

  • First example - same root directory for both to and from file
  • --- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
    +++ outguess-0.2/jpg.c       2009-08-25 16:06:05.242378300 +0200
    
    $ patch -lNp0 <project.patch                        # First example
    
  • second example - different root directory between to and from file
  • --- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
    +++ outguess-0.2-patched/jpg.c       2009-08-25 16:06:05.242378300 +0200
    
    $ cd project_dir; patch -lNp1 <../project.patch     # Second example
    

vbindiff

vbindiff — hexadecimal file display and comparison

vbindiff file1.bin file2.bin               # Compare and view 2 binary files

Miscellaneous

cksum / jacksum / md5sum / sha1sum / sum

cksum file                                              # Generate a CRC similar to CRC32, but in decimal, and with length appended.
cksum -o3 file                                          # Generate a CRC32 checksum (BSD only)
sum file                                                # Generate a checksum
md5sum file                                             # Hash file using MD5
sha1sum file                                            # Hash file using SHA1
jacksum -a crc32 file                                   # Hash using CRC32 (dec)
jacksum -a crc32 -E hexup file                          # Hash using CRC32 (hex)

dog

dog is better than cat. It writes the contents of each give file, URL, or the standard input to standard output.

dtach

dtach simple program that emulates the detach feature of screen. A session in dtach is a single instance in which a program is running under the control of dtach. The program is disassociated from the original terminal, and is thus protected from your original terminal being disconnected for some reason.

echo

echo -e "Some text\n...on 2 lines..."                    # Enable interpretation of backslash escapes (must be quoted!)

expect

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.

iconv

locale encoding conversions

konwert

fancy encoding conversions

mc (Midnight Commander)

mc or Midnight Commander is a powerful file manager working in a shell terminal.
See dedicated page on this wiki.

mimencode

binary file conversion for the mail.

mrxvt

A multi-tab version of rxvt (terminal-emulator).

recode

recode /cl../cr <dos.txt >mac.txt
recode /cr.. <mac.txt >unix.txt
recode ../cl <unix.txt >dos.txt
recode charset1/surface1..charset2/surface2 <input.txt >output.txt
recode /QP.. <qp.txt >normal.txt                                    # To convert quoted-printable text
charset surface
us ASCII (7 bits) /cr Carriage return as end of line (Mac text)
l1 ISO Latin-1 (ISO-8859-1, Western Europe, 8 bits) /cl Carriage return line feed as end of line (DOS text)
EUCJP EUC-JP for Japanese (Unix) / Line feed as end of line (Unix text)
SJIS Shift-JIS for Japanese (Microsoft) /d1 Human readable bytewise decimal dump
ISO2022JP Mail encoding for Japanese (7 bits) /x1 Human readable bytewise hexidecimal dump
u2 UCS-2 (Universal Character Set, 2 bytes) /64 Base64 encoded text
u8 UTF-8 (Universal Transformation Format, 8 bits) /QP Quoted-Printable encoded text

reformime

MIME E-mail reformatting tool

screen

Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.

See also tmux for a better alternatives that supports splitting in panes...

Launch with

screen                                      # Start a new screen session
screen -r                                   # Resumes a detacked screen session
screen -d -r                                # Reattach a session and if necessary detach it first
screen -d -R                                # Reattach a session and if necessary detach or even create it first
screen -D -r                                # Reattach a session. If necessary detach and logout remotely first
screen -D -R                                # Attach here and now, ie:
                                            # - If no session is running, create it
                                            # - Otherwise reattach the session. If necessary detach/logout remotely first

For convenience one may add the following alias to .bashrc:

alias screen=screen -D -R

Useful shortcuts:

key description
C-a c Open a new instance
C-a Space Switch to next instance
C-a d Detach from screen. Reattach later with screen -r
C-a D D Power detach, i.e. detach and logout

byobu is a wrapper around screen, which makes it prettier and more powerful (see this also).

sponge

sponge is part of package moreutils. It can be used to easily edit file in-place:

sed -r '...' FILE | grep ... | sponge FILE                   # Sponge soaks its full input before creating output file

tmux

tmux is actually a mix of screen and Gnome-Terminator. It is a terminal multiplexer: it enables a number of terminals to be created, accessed, and controlled from a single screen. Terminals can be organised in multiple panes in the same window.

Launch with

tmux                 # Start tmux (assume command 'new')
tmux attach          # Start tmux and reattach to previous session

tmux complains if it is launched as an ssh command (error not a terminal), but this is fixed with option -t:

ssh -t server tmux attach     # Launch tmux directly, require forcing terminal emulation

Useful shortcuts:

key description
C-b "

C-b %
C-b o
C-b C-o
C-b ←|→|↑|↓

Split horizontally

Split vertically
Go to next pane
Go to previous active pane
Go to ←, →, ↑ or ↓ pane

C-b q
Display pane id briefly
C-b d
Detach from tmux. Reattach later with tmux attach


Attach automatically

One can configure tmux to attach automatically to an existing session, or create one if none exists:

  1. Add to ~/.tmux.conf
  2. new-session
    
  3. Add to ~/.bashrc
  4. # tmux - always attach (or pass given command and parameters)
    #        This solution requires to have the line "new-session" in ~/.tmux.conf
    function tmux() {
        T=/usr/bin/tmux
        if [ $# -eq 0 ]; then $T attach; else $T "$@"; fi
    }
    

Another solution (but not atomic) that does not require editing .tmux.conf is to use tmux attach || tmux new. This solutions also allows to use named sessions (see [13]).


Zoom on a pane

version 1.8 adds the zoom feature like terminator (see flag -Z to command resize-pane). Here a work-around for previous versions that binds the feature to C-b z (from [14]):

  1. Edit ~/.tmux.conf
  2. unbind z
    bind z run ". ~/.tmux/zoom"
    
  3. Edit ~/.tmux/zoom
  4. #!/bin/bash -f
    currentwindow=`tmux list-window | tr '\t' ' ' | sed -n -e '/(active)/s/^[^:]*: *\([^ ]*\) .*/\1/gp'`;
    currentpane=`tmux list-panes | sed -n -e '/(active)/s/^\([^:]*\):.*/\1/gp'`;
    panecount=`tmux list-panes | wc | sed -e 's/^ *//g' -e 's/ .*$//g'`;
    inzoom=`echo $currentwindow | sed -n -e '/^zoom/p'`;
    if [ $panecount -ne 1 ]; then
        inzoom="";
    fi
    if [ $inzoom ]; then
        lastpane=`echo $currentwindow | rev | cut -f 1 -d '@' | rev`;
        lastwindow=`echo $currentwindow | cut -f 2- -d '@' | rev | cut -f 2- -d '@' | rev`;
        tmux select-window -t $lastwindow;
        tmux select-pane -t $lastpane;
        tmux swap-pane -s $currentwindow;
        tmux kill-window -t $currentwindow;
    else
        newwindowname=zoom@$currentwindow@$currentpane;
        tmux new-window -d -n $newwindowname;
        tmux swap-pane -s $newwindowname;
        tmux select-window -t $newwindowname;
    fi
    


Scroll with mouse wheel

Add to ~/.tmux.conf:

#set-window-option -g mode-mouse on                  # Does not work when ssh from cygwin
set -g terminal-overrides 'xterm*:smcup@:rmcup@'

uuencode, uudecode

Binary file conversion for Unix.

wc

Counts lines, words and character in a file

xargs

For more examples, see combinations with the command find

#By default, xargs cuts at WHITE SPACE and NEWLINE. Use -d '\n' to only cut at NEWLINES
for i in $(locate .pdf); do basename $i; done      # WRONG
locate .pdf | xargs -d '\n' -n 1 basemane          # Correct

Hex Tools

hd / hexdump

ASCII, decimal, hexadecimal, octal dump.

hexdump -e '"%2x"' <myfile>             # Convert myfile into a long hexadecimal string - ! See DOUBLE-QUOTED parameter
hexdump -C <myfile>                     # Canonical hex + ascii display (16 bytes)
hd <myfile>                             # (idem)

See also od and xxd. There are also hex editors, like hexer and hexedit (adapts to width of the terminal!).

od

od - dump files in octal and other formats

od -xc file                                              # Output file in ascii and hex output (2-byte, ! little-endian)
od -t x1 -a --width=32 testvar.sh                        # Output 32-byte at once

See also hd / hexdump and xxd.

xxd

Make a hexdump or do the reverse. This tool is delivered along with vim.

echo 202122 | xxd -r -p                    # Convert hexdump to a binary string
echo -n ' !"' | xxd -p                     # Convert binary string to hexdump - DON'T FORGET -n
echo -n ' !"' | xxd -u -p                  # Convert binary string to hexdump - DON'T FORGET -n (uppercase)

See also hd / hexdump and od.