Linux Commands

References

From The Linux Documentation Project:

• 16.4. Text Processing Commands

File System

ack

ack is a grep-like program specifically for large source trees. It is distributed via package ack-grep. Add a symlink for convenience:

sudo ln -s /usr/bin/ack-grep /usr/local/bin/ack

Features:

• It's blazingly fast because it only searches the stuff you want searched.
• ack ignores most of the crap you don't want to search
• Lets you specify file types to search, as in --perl or --nohtml.
• Color highlighting of search results.
• Uses real Perl regular expressions, not a GNU subset.

# grep pattern $(find . -type f | grep -v '\.svn')
ack pattern                                           # Ack ignore CVS, SVN dir 
# grep pattern $(find . -name '*.pl' -or -name '*.pm' -or -name '*.pod' | grep -v .svn)
ack --perl pattern                                    # Specify file type to search
ack -f --perl > all-perl-files                        # Create rapidly list of files

Supported file types (more here):

    --[no]asm          .asm .s
    --[no]batch        .bat .cmd
    --[no]binary       Binary files, as defined by Perl's -B op (default: off)
    --[no]cc           .c .h .xs
    --[no]cpp          .cpp .cc .cxx .m .hpp .hh .h .hxx
    --[no]css          .css
    --[no]hh           .h
    --[no]html         .htm .html .shtml .xhtml
    --[no]java         .java .properties
    --[no]js           .js
    --[no]make         Makefiles (including *.mk and *.mak)
    --[no]perl         .pl .pm .pod .t
    --[no]php          .php .phpt .php3 .php4 .php5 .phtml
    --[no]python       .py
    --[no]ruby         .rb .rhtml .rjs .rxml .erb .rake .spec
    --[no]shell        .sh .bash .csh .tcsh .ksh .zsh
    --[no]skipped      Files, but not directories, normally skipped by ack (default: off)
    --[no]text         Text files, as defined by Perl's -T op (default: off)
    --[no]vim          .vim
    --[no]xml          .xml .dtd .xsl .xslt .ent

ag

ag (home, github, ack.vim, ag.vim) is the silver searcher, and is actually much better than ack, which in turn is much better than grep ;-). Update: turns out that ripgrep is an even faster alternative.

On recent distribution, install from package:

sudo apt install silversearcher-ag

To install from git (see README.md):

sudo apt-get install software-properties-common
sudo apt-get install -y automake pkg-config libpcre3-dev zlib1g-dev liblzma-dev
git clone https://github.com/ggreer/the_silver_searcher.git
cd the_silver_searcher/
./build.sh
sudo make install

Tips

• Use /b to find whole word matches only, or better -w:

ag "\bWHOLEWORD\b"
ag -w WHOLEWORD

chattr

chattr change file attributes on a Linux file system.

See man chattr for a list of attributes (immutable...)

cp

To copy content of a directory into another:

cp -a /source/. /dest/

This copy recursively and preserve attributes -a, and copy the content with special syntax source/. with trailing dot.

To copy a directory recursively, create hard links instead of new copies (but creates new directories, not links):

cp -lr <src> <target>

See #cpg for a patch that adds a progress bar.

cpg

cpg is a patched version of cp that adds a progress bar, advcpmv(github).

cpg -ag src1 dst2              # Copy recursively with progress

Alternatives are to use rsync --progress (or rsync -P), or #gcp.

dd

Convert and copy a file.

dd returns current copy status when sent an HUP signal. Type in another shell:

while :; sleep 5; do killall -s SIGHUP1 dd; done

This will force dd to update its status every 5sec.

Transfer speed is highly dependent on the block size! So make sure to use -bs=16M or something like that:

 dd if=/dev/sda1 of=backup-sda1 bs=16M

Other recommended values for -bs are: 1MB [1], 64k [2] [3].

Note: to copy a whole file or device cat or pv are better alternatives (see Linux Disk Management). cat is at least faster than dd with default parameters [4], and pv looks for optimal block size to get highest speed.

dd if=file1 of=file2             # slow
cat file1 > file2                # fast
dd if=file1 of=file2 bs=16M      # faster (maybe)
pv file1 > file2                 # fastest

Interesting links:

• dd-opt, an utility to find optimal value for bs parameter.
• A complete blog post about Tuning dd block size
• Discussions on stackexchange.com, also suggesting that cat could be used too.
• ... but dd still has many other uses that cat does not provide.
• ... for instance, create sparse files, and mount it as filesystem loopback:

# Create sparse file
dd of=sparse-file bs=1 count=0 seek=10           # SAME AS: truncate -s 10GB sparse-file
# Mount it
loop=`losetup --show -f sparse-file`
mkfs.ext4 $loop
mkdir myloop
mount $loop myloop

• Useless Use Of dd

# Write myfile.iso to a USB drive
# !!! with -a, cp may try to create a new block device!
cp myfile.iso /dev/sdb

# Rip a cdrom to a .iso file
cat /dev/cdrom > myfile.iso

# Create a gzipped image
# !!! w/o redirection, gzip may skip non-regular file /dev/sdb
gzip -9 < /dev/sdb > /tmp/myimage.gz

entr

entr runs arbitrary commands when files change.

For instance:

find -name *.c | entr make

fatsort

fatsort sorts directory structures of FAT filesystems. Many MP3 hardware players don't sort files automatically but play them in the order they were transmitted to the device. FATSort can help here.

sudo fatsort /dev/sdb1

fd

fd is a simple, fast and user-friendly alternative to 'find'.

fd
fd app
fd sh
fd sh --type f
fd -e md
fd -e md --exec wc -l
fd --hidden sample

find

Tip: Consider using fd as simpler and faster alternative.

See here for further examples on how to combine find and xargs.

FREQUENT PITFALLS!

• ESCAPE THE SEMI-COLON — The semi-colon ; must be escaped in Bash !
• USE - OR + WITH NUMERICAL VALUES — for instance -mtime 1 means exactly one day ago, whereas -mtime -1 means within the last day.

find . -name "*.jpg"                                     # find files/directories whose name matches specified pattern
find . -path "*/cur/*"                                   # find files/directories whose path spec matches specified pattern
find . -type d -name "cur"                               # find only directories whose name matches specified pattern
find . -exec echo '{}' \;                                # semi-colon must be escaped!
find . -exec echo one '{}' \; -exec echo two '{}' \;     # to execute several commands on one match
find . -name 'dirname' -prune        -o -iname '*.txt' -print
                                                         # Find all text files, but exclude any directory named dirname
find . -name 'dirname' -prune -false -o -iname '*.txt'   # ... idem (using -false, but no longer -print)
find . -path './path'  -prune -false -o -iname '*.txt'   # Find all text files, but exclude path ./path
find . -iname '*.zip' -exec sh -c 'unzip -l "$0" | egrep -o " [0-9]+ file"' '{}' \;
                                                         # How to use pipe in -exec.
find www -name '*.pdf' -printf %P\\n                     # Print PDF files in www/ but remove the leading www/ in output (%P)

Accelerating find:

find / -mount -name "*splash*"                           # Only search on CURRENT file system (same as find -xdev)

Use also xargs to accelerate find:

find . -name \*.jpg | xargs -r echo                      # much faster than find -exec
find . -name \*.jpg | xargs -rn 2 echo                   # limit #args passed to echo to 2 max.
find . -name \*.jpg -print0 | xargs -r0 ls               # Use NULL separator if files contains space or special char !!!

# find -exec command \; and xargs -i are equivalent
find . -name "*.txt" -exec echo '{}' is found. \;        # ONE at a time, but {} can be anywhere - and MULTIPLE times
find . -name "*.txt" -print0 | xargs -r0 -i echo '{}' is found.  # ... same, with xargs
# find -exec command {} + and xargs are equivalent
find . -name "*.txt" -exec echo The files are '{}' +     # FASTER - much like xargs. Only append AT THE END!
find . -name "*.txt" -print0 | xargs -r0 echo The files are   # ... same with xargs

# With mv, use -t DIRECTORY to give destination directory first
find . -size +100M -print0 | xargs -0 mv -t bigfiles/    # -t DIR inverts usual SOURCE... DESTINATION order
find . -size +100M -exec mv -t bigfiles/ {} +            # ... idem, without xargs

FSlint

FSlint is a utility to find and clean various forms of lint on a filesystem. I.E. unwanted or problematic cruft in your files or file names. For example, one form of lint it finds is duplicate files.

gcp

A variant of cp, but with progress bar:

gcp -pR dir1 .

If complains about missing dbus:

dbus-launch gcp -pR dir1 .

An alternative is to use a patched version of cp.

getfattr / setfattr

getfattr and setfattr gets/sets extended attributes of filesystem objects.

!!! These tools are really misleading, and many attributes are hidden. Usually you need to list the attributes explicitly !!!

getfattr -d file                      # This SUPPOSEDLY returns *ALL* attributes, but actually *MANY* missing
getfattr -m - file                    # This MIGHT return more attributes
getfattr -n system.ntfs_attrib file   # View ntfs-3g special attributes
getfattr -d -e hex file               # View attribute values in HEX

grep / zgrep

grep -Rsl PATTERN [FILE]            # Recursive, no error output, only list filename
grep BASIC-REG-EXP-PATTERN [FILE]   # Use classic regexp (like "dma\|DMA")
egrep EXT-REG-EXP-PATTERN [FILE]    # Same as grep -E. Use extended regexp (like "dma|DMA")
fgrep FIXED-STRINGS-REG-EXP [FILE]  # Same as grep -F. Pattern is a list of strings to match.
grep -n PATTERN [FILE]              # Print matched line numbers.
grep -- "-s" [FILE]                 # Search for text "-s"
grep -e "-s" [FILE]                 # Search for text "-s" - alternative solution
grep -R -include=*.in PATTERN *     # Search recursively through folders, limiting to files matching pattern "*.in"
grep -R PATTERN *.in                # Idem, but matching pattern "*.in" also applies to folders.
grep -o PATTERN [FILE]              # (--only-matching) Print only the matched (non-empty) parts of a matching line.

zgrep works like grep, but also accepts compressed (.gz) files.

zgrep PATTERN *.txt *.txt.gz        # Search in .txt file (compressed or not)

gzip / gunzip / zcat

gzip -c FILE                        # Compress FILE to stdout
gunzip -c FILE                      # Uncompress FiLe to stdout
zcat FILE                           # Equivalent to gunzip -c FILE

ls

Listing directory content:

ls --full-time               # To list files full date & time
ls -d directory-name         # List directory entries, not content
                             #  eg. ls -d /etc/rc*
ls -lS | head                # List biggest files first - print only first 10 entries
ls -lt | head -n 20          # List most recent files first, print only 20 first entries

lsattr

lsattr list file attributes on a Linux second extended file system.

sudo lsattr

---------------- ./memtest86+.elf
---------------- ./System.map-3.13.0-39-generic
---------------- ./vmlinuz-3.13.0-39-generic
---------------- ./initrd.img-3.13.0-39-generic
----------I----- ./grub
---------------- ./abi-3.13.0-39-generic
---------------- ./memtest86+_multiboot.bin
---------------- ./memtest86+.bin
---------------- ./config-3.13.0-39-generic
---------------- ./lost+found

ntfsundelete

ntfsundelete recover a delete file from an NTFS volume. Does not scan the disk.

See [5], [6].

par2

par2 is a very useful program for creating and using PAR2 files to detect damage in data files and repair them if necessary.

From the manpage, using an example 800MB test.mpg:

par2 create test.mpg        # Will create *.par2 files, for a total of roughly 40MB (5% redundancy)
par2 verify test.mpg.par2   # to verify integrity
par2 repair test.mpg.par2   # to repair

*.par2 files stores error correction informations, in a number of recovery blocks. Repair is possible as long as there are enough recovery blocks available (on Usenet era, it was useful to have *.par2 files of increasing size such that to allow users to minimize bandwidth usage).

par2 is particularly useful to protect MP3 files on USB sticks (that have frequent FS failure over time):

par2 create check.par2 *.mp3   # Generate recovery blocks (5% redundancy)
par2 verify check.par2         # Verify
par2 repair check.par2         # Repair

To increase redundancy:

par2 create -r10 check.par2 *.mp3   # Generate recovery blocks (10% redundancy)

Note: to simulate file corruption, we can use:

# Simulate 25kB corruption, from 10k-th byte.
dd if=/dev/zero of=output.mp3  bs=1024 count=25 seek=10 conv=notrunc

progress

progress [7] watches progress of other utilities like cp, mv, dd, tar.

# display estimated I/O throughput and estimated remaining time for on going coreutils commands
progress -w       

# Start an heavy command and monitor it with -m and $!
tar czf images.tar.gz linuxmint-18-cinnamon-64bit.iso CentOS-7.0-1406-x86_64-DVD.iso CubLinux-1.0RC-amd64.iso | progress  -m  $!

pv

pv monitors the progress of data through a pipe. It can also be used to copy files or entire volume very rapidly.

For instance:

pv </dev/sda >/dev/sdb

Also over the network using nc:

# On Host A (receiver):
nc -l 2222 > /dev/sda
# On Host B (sender):
pv </dev/sda | nc hosta 2222                # 111MB avg throughput on direct link (cross-cable), SSD hard disks

rdfind

rdfind finds duplicate files, very rapidly.

rdfind /foo/dir /bar/dir                       # Find duplicates in give dir and tell what could be gained
rdfind -makehardlinks true /foo/dir /bar/dir   # Remove dups by creating hardlinks

rg

rg or ripgrep is an extremelly fast alternative to grep (in fact even faster than ag the silver-searcher).

See ripgrep on GitHub for ripgrep.deb package.

rsync

Perfect copy over network (see this page for details)

#If needed, pre-activate sudo on remote system. Flag -t required to solve 'sudo: no tty present and no askpass program specified'
#
# Also, this requires the following line in /etc/sudoers:
#
#    Defaults     !tty_tickets
#
stty -echo; ssh -t user@server sudo -v; stty echo

sudo rsync -aHAXS --delete --rsync-path "sudo rsync" --numeric-ids -h -v --exclude='lost+found' user@server:/remote/path  /local/path

More basic use of rsync:

sudo rsync -aP src1 dst2           # Copy with progress

scalpel

scalpel recover files using a header / footer database (see [8]).

Install scalpel, then edit /etc/scalpel/scalpel.conf to select which file types to recover (based on header/footer).

sudo apt-get install scalpel
sudo vi /etc/scalpel/scalpel.conf

Here we enable jpg and mp3 (see [9] for more):

# GIF and JPG files (very common)
#	gif	y	5000000		\x47\x49\x46\x38\x37\x61	\x00\x3b
#  	gif	y 	5000000		\x47\x49\x46\x38\x39\x61	\x00\x00\x3b
 	jpg	y	200000000	\xff\xd8\xff\xe0\x00\x10	\xff\xd9
        jpg     y       200000000       \xff\xd8\xff\xe1                \xff\xd9 
# MP3
       mp3     y       8000000 \xFF\xFB??\x44\x00\x00
       mp3     y       8000000 \x57\x41\x56\45            \x00\x00\xFF\
       mp3     y       8000000 \xFF\xFB\xD0\            \xD1\x35\x51\xCC\
       mp3     y       8000000 \x49\x44\x33\
       mp3     y       8000000 \x4C\x41\x4D\x45\

Start scalpel:

sudo scalpel /dev/sda1 -o output_dir      # output_dir must *not* be on disk being scanned

tar

tar archives and restores timestamps. When run as root, tar will also restore ownership and permissions.

tar -cvzf archive.tgz somedir/                             # Archive & GZIP compress the directory somedir/ (recursively)
tar -cvjf archive.tar.bz2 somedir/                         # ... Use BZIP2 insteadl
tar -cvzf archive.tgz --exclude=notme somedir/             #  ... exclude any file matching name
tar -cvzf archive.tgz --exclude=somedir/not/me somedir/    #  ... exclude file at specific location

tar -xvzf archive.tgz                                      # Extract archive (will create somedir/)
tar -xvzf archive.tgz -C targetdir                         # ... to specified directory
tar -tvzf archive.tgz                                      # List the content (equiv. of tar --list -vzf ...)

tar cf - --sort=name --owner=root:0 --group=root:0 \
  --mtime='UTC 2019-01-01' . | gzip -n > invariant.tgz     # Create a stable archive w/o timestamps, etc 
                                                           # https://stackoverflow.com/questions/32997526/how-to-create-a-tar-file-that-omits-timestamps-for-its-contents

Troubleshootigng

• On windows, getting errors like tar: ....myfile.txt: file changed as we read it.

No fix. We silent the error with the following script:

nice_tar()
{
    # Bloody tar returns a warning about modified dir on windows. Must silent the warning.
    tar "$@"
    RC=$?

    [ $RC -eq 0 -o $RC -eq 1 ] && return 0 || return $RC
}

testdisk / photorec

testdisk scans and repairs disk partitions. photorec recovers lost files from harddisk, digital camera and cdrom (including audio and photo files).

photorec /log /d output_dir /dev/sda1

ug

ug or ugrep is an extremelly fast alternative to grep (in fact even faster than ag the silver-searcher, and comparable speed with ripgrep, but with some more powerful search capabilities).

See ugrep on GitHub (also nice instructions for setup on Vim).

Miscellaneous

badblocks

An utility to detect bad sectors on USB flash drive or so.

dosfslabel

Change FAT16/FAT32 disk label

sudo dosfslabel /dev/sdc1 UBUNTULIVE

mbr

Install MS-like Master Boot Record (see [10])

sudo apt-get install mbr
install-mbr -i n -p D -t 0 /dev/sda        # Fix MBR on /dev/sda
                                           # -i interrupt n = none (do not display a MBR prompt)
                                           # -p partition D = boot the partition with the bootable flag set
                                           # -t timeout 0 = do not wait to boot
dd if=/dev/sda of=opensource.mbr bs=512 count=1
install-mbr -i n -p D -t 0 opensource.mbr  # Create a file opensource.mbr containing the generated MBR

mkfs

Format disk utility

sudo mkfs.vfat -F 32 -c /dev/sdc1

ms-sys

Install MS-compatible Master Boot Record (similar to FIXMBR on WinXP rescue console, see [11])

# Get ms-sys from http://ms-sys.sourceforge.net/#Download
sudo apt-get install gettext
make
sudo make install
sudo ms-sys -m /dev/sdb

relocntfs

relocntfs (see [12]) is more or less the equivalent of the FIXBOOT command in the Windows Recovery Console. It is among other available on the Trinity Rescue Kit. For instance, to fix the boot record of a windows partition at /dev/sda2 issue the commands:

relocntfs /dev/sda2        
relocntfs -w /dev/sda2                    # To actually write the new boot record

stat

display file or file system status (like file access time, creation time, modification time...)

tree

List the contents of directories in a tree-like format.

Network

autossh

See Autossh.

dhclient

dhclient -r eth1        # Release DHCP lease on eth1
dhclient eth1           # Obtain fresh ip on eth1

dig

dig stands for domain information groper. It replaces the deprecated nslookup. It comes with package dnsutils:

sudo apt-get install dnsutils

To get ip address of a given host:

dig +short google.com | head -n 1      # In case host has multiple address

See also getent hosts or hostname.

To do a reverse DNS lookup:

• Use option -x of dig command:

dig +short -x 82.78.227.176
# 176.176-191.227.78.82.in-addr.arpa.
# ivorde.ro.

• Or query record PTR in in-addr.arpa domain:

dig +short ptr 176.227.78.82.in-addr.arpa.
# 176.176-191.227.78.82.in-addr.arpa.
# ivorde.ro.

getent

Use getent hosts to get ip address of a given host:

getent hosts google.be
getent hosts google.be | awk '{ print $1; exit }'   # To get first ip address only

Note getent may succeed where dig fails (for instance host in /etc/hosts file or hosts with .local suffix).

host

Use host to do a reverse DNS lookup, i.e. get hostname from a given IP address:

host 123.45.67.89

host can also do a regular DNS lookup and fetch alias names:

host -t a www.facebook.com
# www.facebook.com is an alias for star.c10r.facebook.com.
# star.c10r.facebook.com has address 173.252.100.27

hostname

Use hostname -I to get local host ip address:

hostname -I
hostname -I | awk '{ print $1 }'         # In case host has multiple interfaces / ip addresses

ifconfig

See Linux networking.

ip

See Linux networking.

iwconfig

See Wifi.

mail

mail — send and receive mail.

# Using local SMTP
echo my message body here | /usr/bin/mail -s"My subject here" my_email@myserver.org

# Using ISP SMTP - no auth
echo "my message body here" | mail -S 'smtp=smtp.myisp.org:25' -S 'from=first.last@myisp.org' -s 'My subject here' my_email@myserver.org

multitee

multitee sends multiple inputs to multiple outputs.
Check this page.
Original is here, or can also be found here on Debian.

Here a patch to build it on Cygwin. The patch ports calls to BSD signal.h API (struct sigvec, sigvec(), sigmask(), sigblock()...) to the POSIX API (struct sigaction, sigaction(), sigprocmask(), sigfillset()...):

$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin

Example of use:

$ multitee 0-1,4,5 4> foo 5> bar             # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7                 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0   # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1                           # same as 'socat - FD:3'

netcat or nc

TCP-IP swiss army knife (equivalent of the telnet program. Check wikipedia:netcat. Also known as command nc).

See also Using Netcat (Linux Tips).

Using as telnet-like

nc www.immie.org 22                               # Connect to SSH port
nc www.immie.org ssh                              # idem, using port name
nc -v www.immie.org ssh                           # idem, verbose

Use nc also to test whether some ports are open, a bit like nmap. For instance:

nc -vz www.immie.org 25                           # See if SMTP port is open
# www.immie.org [91.134.134.85] 25 (smtp) open
nc -vz www.immie.org 80                           # idem, with http port

Using TCP in multi-connect

Netcat will quit as soon as the first connection terminates. Use option -k to keep the connection alive:

# On DESTINATION host
nc -l -k 6666                                 # Keep connection alive
# On SOURCE host
echo foo | nc griffin 6666
echo bar | nc griffin 6666

Using UDP in multi-connect

Note that When nc is listening to a UDP socket, it 'locks on' to the source port and source IP of the first packet it receives. [13]. So one must give the source port (with option -p) on the SOURCE host as well, or second and next transmissions will be ignored.

# On DESTINATION host
nc -l -u 6666
# On SOURCE host
echo foo | nc -u -w 1 -p 6665 griffin 6666    # Enforce source port 6665. Option -w 1 to timeout after 1 sec
echo bar | nc -u -w 1 -p 6665 griffin 6666

netstat

See Linux networking.

nmap

nmap is the network exploration tool and security / port scanner.

nmap localhost                    # Scan and print all open ports on 'localhost'
nmap -sP hostname                 # Simply test whether 'hostname' is accessible
nmap -sP 192.168.1.0/24           # Scan all local network with mask 192.168.1.0/24 (sudo: show mac address)

Detect host availability

One can use nmap in a bash script to test whether a given host is accessible, but it turns out that ping is faster:

# Using nmap
nmap -sP $1 2>/dev/null | grep -q "1 host up"    # Too slow, does not always detect hosts

# ... or ping (faster)
if [ "$HOST_MACH" = "cygwin" ]; then
    ping $1 1 1 2>/dev/null | grep -q "1 packets received"
else
    ping -c 1 $1 2>/dev/null | grep -q "1 received"
fi

Detect all Raspberry pi on the network

sudo nmap -sP 192.168.1.0/24 | awk '/^Nmap/{ip=$NF}/B8:27:EB/{print ip}'
sudo nmap -sP 192.168.1.0/24 | grep -B2 Raspberry

Various scan

sudo nmap -A -T4 myserver.org         # Quick port scan on myserver.org
sudo nmap -A -T4 -p22 myserver.org    # Get SSH banner from server

nmcli

See Linux networking.

nm-tool

nm-tool is an utility to report NetworkManager state and devices.

nm-tool                   # Print IP addresses, DNS servers, etc.

/etc/init.d/nscd

This is not really a command, but the init service Name Service Cache Daemon.

sudo /etc/init.d/nscd restart                # To restart daemon and flush DNS cache

nslookup

nslookup is a program to query the DNS database. It is available on Linux and Windows, but on Linux is is deprecated by dig.

nslookup 8.8.8.8        # Reverse DNS lookup

rpcinfo

rpcinfo reports RPC information

rpcinfo -p              # Probe the portmapper on host, and print a list of all registered RPC programs

socat

Command-line utility that establishes two bidirectional byte streams and transfers data between them ([14]). socat is the more powerful version of netcat. Check the homepage. And also this page on Yobi for examples on how to use socat to bypass a proxy.

# All these commands listen on localhost:53805, and forward to server:53806
socat TCP4-LISTEN:53805                TCP:server:53806      # Listen, and close as soon socket closes
socat TCP4-LISTEN:53805,reuseaddr,fork TCP:server:53806      # Idem, but fork so that listener continues after socket closes

socat -v ...                                                 # Send txfed data to stderr, with some formatting (text format)
socat -x ...                                                 # ... idem in hex format (can be combined with -v)
socat -ly                                                    # Writes messages to syslog instead of stderr; severity as defined with option -d

socat can easily tunnel connections:

socat -ly 'TCP4-LISTEN:143,reuseaddr,fork' PROXY:ton.imap.server:143|TCP:134.27.168.36:8080

socat can also be used as SSH ProxyCommand:

ProxyCommand socat -ly - 'PROXY:%h:%p|TCP:proxy.server:8080'
# Using socat v2.0.0 beta
ProxyCommand socat -ly - 'PROXY:%h:%p,proxyauth=user:pass|SSL,verify=0|PROXY:my.server:443,proxyauth=user:pass|TCP:big.brother.proxy:8080'

socat can be easily used as a replacement of telnet:

socat tcp:<host>:<port> -       # <port> can be a port number or service name (telnet,imap...)

To establish UDP transfer, use:

socat UDP-RECV:[port] STDOUT               # On destination host
socat STDIN UDP-DATAGRAM:[host]:[port]     # On source host

QUESTION TO MR DOEGOX:
- WHY -s in SED?
- Where does it get info on % for socat?

$ socat echo -
$ socat echo STDIO
$ socat echo STDOUT%STDIN
$ socat echo STDIO%STDIO
$ socat echo -%-
$ socat echo FD:1%FD:0
$ socat echo 1%0

/usr/local/bin/socat TCP:127.0.0.1:1234 -%EXEC:"awk 'BEGIN{print \"BANNER\";fflush()}/BANNER/{next}//{print;fflush()}'"

Testing socat:
- /usr/local/bin/socat - TCP-LISTEN:1234
- /usr/local/bin/socat TCP:127.0.0.1:1234 EXEC:"/bin/sed -s 's/foo/bar/g'"%EXEC:"/bin/sed -u 's/toto/tata/g'"

  First open a port for listening and connect it to STDIO
  In another shell, connect and do the black magic...


reference:
- http://eros.ph0k.eu/~nizox/twin/V2/modules/socat-1.6.0.0/EXAMPLES
- http://www.rschulz.eu/2008/09/ssh-proxycommand-without-netcat.html
		$ exec 3<>/dev/tcp/zeus.yobi.be/22;(cat <&3 & );cat >&3
		SSH-2.0-OpenSSH_5.1p1 Debian-3
- http://thesmithfam.org/blog/2006/05/23/bash-socket-programming-with-devtcp-2/
- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464
- http://www.mtu.net/~engstrom/ssh-proxy.php



ATTEMPT 3:
Window 1: socat -v -%'EXEC:/home/beq06659/.ssh/mys2.sh' TCP-LISTEN:1234
Window 2: /usr/local/bin/socat -ly -v -%"EXEC:sed -unf /home/beq06659/.ssh/sedy" "PROXY:ftp.noekeon.org:22|TCP:localhost:1234"

File mys2.sh:
	#! /bin/bash
	echo -e "HTTP/1.1 200 Connection established\r\n\r\n"
	cat
File sedy:
	1 s/^/SSH-2.0-OpenSSH_5.1\n/p; /SSH-2.0-OpenSSH_5.1/d; /SSH-2.0-OpenSSH_5.1/! p

LESSONS LEARNED:
- READ THE CHANGES DOC, especially for BETA SW
- SOCKET = bidir --> a filter = 4 descriptors
- PIPE = unidir --> a filter = 2 descriptor (stdin/stdout)

ss

See Linux networking.

wavemon

wavemon is a top-like monitoring tool for wifi connection.

To install:

sudo apt install wavemon

To use:

wavemon

wget

wget URL
wget URL --content-dispostion   # If download gets wrong name, use this option first
wget URL -O filename            # ... otherwise use this to rename download to 'filename'

whois

Find CIDR for www.facebook.com [15]:

host -t a www.facebook.com
# www.facebook.com has address 69.171.228.40

whois 69.171.228.40 | grep CIDR
# CIDR:           69.171.224.0/19

Internet

httrack

httrack is a offline browser; it copies websites to a local directory.

# Example: checking out progit boot
# -d                       stay on same principal domain
# -w                       mirror web sites
# -O /var/www              path for mirror/logfiles+cache
# -I0                      don't make an index
# -%v                      display on screen filesnames downloaded
# -P 127.0.0.1:8118        Proxy settings
# '+progit.org/figures/*'  +filter
# '-progit.org/book/*/*'   -filter
httrack http://progit.org/book/ -d -w -O /var/www -I0 -%v '+progit.org/figures/*' '-progit.org/book/*/*' [-P 127.0.0.1:8118]
sed -ri \"s_'http://progit.org/book'_'index.html'_\" /var/www/progit.org/book/*.html

# Example 2
# -W                       mirror web sites, semi-automatic (ask question)
httrack http://www.cplusplus.com/ -W -O /var/www -%v '-www.cplusplus.com/forum/*' '-www.cplusplus.com/src/*' '-www.cplusplus.com/member/*' [-P 127.0.0.1:8118]

# Example 3 - Use different path for cache/log files
#  -O /var/www,/tmp/httrack   path for mirror,path cache and logfiles
httrack http://www.argospress.com/Resources/systems-engineering/index.html -d -O /var/www,/tmp/httrack -I0 -%v [-P 127.0.0.1:8118]
sed -ri '/wms.assoc-amazon.com/{N; d}' /var/www/www.argospress.com/Resources/systems-engineering/*.htm

# Example 4 - Limit depth
#  -r2                     set mirror depth to 2 (i.e. given address + 1st links)
httrack http://powerman.name/doc/asciidoc -r2 -w -I0 [-P 127.0.0.1:8118]

Serial

stty

stty changes and prints terminal line settings. It usually comes pre-installed.

stty -F /dev/ttyUSB0 -a         # Print all settings in human-readable form
stty -F /dev/ttyUSB0 --save     # Print all settings in stty-readable form

setserial

setserial get/set Linux serial port information.

See for instance [16] for usage.

Package Management

dpkg

Package manager for Debian / Ubuntu. See Package Management#Ubuntu / Debian

rpm

See Package Management#RPM's

System

dmidecode

dmidecode is a tool to decode DMI (SMBIOS) tables:

sudo dmidecode

...
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
	Vendor: Dell Inc.
	Version: A19
	Release Date: 12/21/2009
	Address: 0xF0000
	Runtime Size: 64 kB
	ROM Size: 1728 kB
	Characteristics:
		ISA is supported
		PCI is supported
		PC Card (PCMCIA) is supported
		PNP is supported
		BIOS is upgradeable
		BIOS shadowing is allowed
...

inxi

inxi is yet another way to get information on hardware:

inxi -Gx           # as current desktop user, don't use sudo!
# Graphics:  Card: Intel 3rd Gen Core processor Graphics Controller bus-ID: 00:02.0 
#            X.Org: 1.15.1 driver: intel Resolution: 1600x900@60.0hz 
#            GLX Renderer: Mesa DRI Intel Ivybridge Mobile GLX Version: 3.0 Mesa 10.1.0 Direct Rendering: Yes

loadkeys

loadkeys loads keyboard translation tables.

Use loadkeys to change the keyboard layout in console mode (i.e. Ctrl-Alt-F1 console).

sudo loadkeys be                   # Load Belgian layout

lscpu

lscpu                              # Display information about the cpu architecture

lsmem

lsmem                              # List all ranges of available memory

lsmod / modinfo / modprobe

lsmod                              # Show the status of modules in the Linux Kernel
modinfo iwlagn                     # Show information about a Linux Kernel module
modprobe -r iwlagn                 # Remove module from the Linux Kernel
modprobe iwlagn 11n_disable=1      # Add module to the Linux Kernel, with specific parameters

lspci

lspci                              # List all PCI devices
lspci -v                           # List all PCI devices (verbose)

newgrp

newgrp log in to a new group

 
newgrp - <group>       # Start a new session with <group> as primary group. - means to reinitialize environment.
umask 002              # Make sure all files / directories we create have group access

To run newgrp in a script, use the following construction [17]:

newgrp groupb << END
    touch "myscript_output.txt"
END

Or better yet use the sg command instead.

pidinfo

Find the process ID of a running program

pidof apache2                      # Find all pid of apache2

pgrep

Prints PID of the process matching given pattern. Never match pgrep itself!

ps -fp $(pgrep autossh)            # Much better that "ps -aef | grep autossh", which match grep process itself

pkill

Send a signal (default TERM) to process matching given pattern. Never match pkill itself!

pkill -0 autossh                   # Test if process "autossh" is running

ps

ps -ef                             # All processes            - full format

ps afl                             # Proc w ttys, all users   - long + forest
ps flx                             # Proc w/o ttys            - long + forest
ps aflx                            # All processes, all users - long + forest

ps au                              # Proc w ttys, all users   - user output
ps ux                              # Proc w/o ttys            - user output
ps aux                             # All processes, all users - user output

ps axs                             # All process, with signals

pv

pipe viewer — terminal-based tool for monitoring the progress of data through a pipeline

pv -cN source < linux-2.2.20.tar.bz2 | bzcat | pv -CN bzcat | gzip -9 | pv -cN gzip > linux-2.2.20.tar.gz

   source: 4.48MB 0:02:01 [9.84kB/s] [========>                        ] 29% ETA 0:04:44
    bzcat:   26MB 0:02:01 [ 192kB/s] [               <=>                               ]
     gzip: 5.58MB 0:02:01 [  64kB/s] [                         <=>                     ]

savelog

savelog saves, rotates and compresses log files. This is similar to logrotate cron job.

savelog -n ~/log/git-fetch.log

sg

sg executes command as different group ID.

umask 002                     # Ensure that all files written will be group accessible
sg - <group> -c "mkdir foo"   # Run commands as group <group>

The following construction restart the current script under a different group (from [18]):

if ! [ "${SBREADY:=false}" = true -o $(id -ng) = ${NEEDGRP:=wheel} ] ; then
    export SBREADY=true
    exec sg $NEEDGRP "$0" "$@"
fi

sudo / gksudo

See also Ubuntu's page at [linux reference script security sudo ubuntu].

ls | sudo tee /root/somefile                # Instead of redirection sudo ls > /root/somefile
ls | sudo tee -a /root/somefile             # ... same but append instead
sudo sh -c "ls > /root/somefile"            # Yet another way to do redirection with sudo
sudo !!                                     # Repeat last command with root privilege
sudo -i                                     # Get a root *login* shell
sudo su -                                   # ... another solution
sudo -s                                     # Root shell but keeping current environment
sudo su                                     # ... another solution
sudo -k                                     # Reset sudo password timeout

gksudo is the graphical equivalent of sudo:

gksudo gedit /etc/fstab

showkey

showkey examine the codes sent by the keyboard (linux kernel only - for X keyboard events, see xev!).

showkey -k                                  # Dump key codes (default)
showkey -s                                  # Dump scan codes

timeout / timelimit

timeout / timelimit are small binary to run a command with a given time limit. More exactly they send a given signal to the child process after some specified time. See also Perl - kill on ALARM signal).

timeout -15 30 mplayer dvd://              # Show 30s of DVD movie, then kill mplayer with a TERM signal

Memory related

free

Display the amount of free and used memory (free -m to get info in MB)

free
free -m       # Show in megabyte

/proc/meminfo

All memory usage information

cat /proc/meminfo

vmstat

Summary of memory usage

vmstat

Miscellaneous

htop

an improved top command

lsb_release

lsb_release -a prints version information for the Linux release you're running.

Another option (also working on Debian), is to use cat /etc/*-release.

mkfifo

make FIFOs (named pipes)

pv, pipeview, pipebench

monitor the progress of data through a pipe

strace

trace system calls and signals

tee

read from standard input and write to standard output and files

uname

uname -a prints all system information, include machine name, kernel name & version.

watch

Execute a program periodically, showing output full screen

watch -n 0.5 myprog arg1 arg2          # Wait 0.5s between run
watch -cn 0.5 myprog arg1 arg2         # ... same with color
watch -cn 0.5 myprog \"arg w/ space\"  # BEWARE! arg must be "double" quoted
watch -cn 0.5 myprog ${*@Q}            # Quoted args, in case args are user defined in a script

User / Group Administration

groupadd / addgroup

Use groupadd, addgroup to create a new group. groupadd is the low-level utility. addgroup is the Debian more friendly version.

sudo groupadd groupname                      # Create a new group groupname
sudo groupadd -g 123 groupname               # ... and specify gid of the new group
sudo addgroup groupname
sudo addgroup --gid 123 groupname

useradd / adduser

Use useradd to create new user. useradd is the low-level utility. adduser is the Debian more friendly version.

sudo useradd username                        # Create a new user username
sudo useradd -s /bin/bash username           # ... specify the deafult shell in use for the user
sudo useradd -s /bin/bash -m username        # ... and create user home directory (from /etc/skel)
sudo useradd -u 1234 username                # Create a new user username with uid 1234
sudo useradd -g GRP1 -G GRP2,GRP3 username   # Specify primary and supplementary groups
sudo adduser username                        # Create a new user username
sudo adduser --system username               # Create a new system user (in the system uid range)
sudo adduser --shell /bin/bash username      # Use /bin/bash as default login shell

userdel / groupdel / deluser

Use userdel, groupdel, deluser to remove an existing user /group. deluser is the Debian more friendly version

sudo userdel username                        # Remove existing user username
sudo groupdel groupname                      # Remove existing group groupname
sudo deluser username                        # Remove existing user username
sudo deluser --group groupname               # Remove existing group groupname

usermod

Use usermod to modify an existing user

sudo usermod -g GRP1 username                # Modify the primary group of user username
sudo usermod -g GRP1 -G GRP2,GRP3 username   # ... and also the supplementary groups

gpasswd

Use gpasswd to administer /etc/group and /etc/gshadow

sudo gpasswd -a $USER fuse                   # Add self to group 'fuse'

To make the new group effective:

loginctl terminate-user $USERNAME            # This will FORCE LOGOUT !!!
su - $USERNAME                               # New shell with new groups

X

disper

TO install:

sudo add-apt-repository ppa:wvengen/ppa
sudo apt-get update
sudo apt-get install

Some examples of use:

disper -s               # Only enable primary screen (laptop screen)
disper -S               # Only enable secondary screen (external monitor)
disper -d auto -e       # Auto-detect + enable extended mode
disper -d auto -c       # clone mode

xbindkeys / xbindkeys-config

• xbindkeys is a program that allows you to launch shell commands with your keyboard or your mouse under the X Window System. It links commands to keys or mouse buttons, using a configuration file. It's independent of the window manager and can capture all keyboard keys (ex: Power, Wake...).
• xbindkeys-config is an easy to use gtk program for configuring Xbindkeys (see [19].

xclip

xclip is a command line interface to the X11 clipboard. It can also be used for copying files, as an alternative to sftp/scp, thus avoiding password prompts when X11 forwarding has already been setup. Check this guide.

xdpyinfo

Display information utility for X, like the dimension of the display (pixels vs millimeters) and the resolution (DPI):

xdpyinfo | grep dimensions               # Return dimension of the desktop (size in pixels and millimeters)
xdpyinfo | grep resolution               # Return current screen resolution in DPI (dots per inch)

XSel / XSelection

XSel is a command-line program for getting and setting the contents of the X selection and context-menu clipboard. Normally the former is only accessible by manually highlighting information and pasting it with the middle mouse button, and the latter is only accessible by using context-menu cut/copy/paste command.

xselection is an equivalent package on OpenSUSE.

On Cygwin, one has to use getclip, putclip or device /dev/clipboard.

xsel -p                                 # To get content of PRIMARY buffer (X selection)
xselection PRIMARY                      # (equivalent command for xselection)
xsel -b                                 # To get content of CLIPBOARD (context-menu clipboard)
xselection CLIPBOARD                    # (equivalent command for xselection)
echo sometext | xsel -p                 # To set PRIMARY buffer
echo sometext | xselection PRIMARY -    # (equivalent command for xselection)
xselection PRIMARY sometext             # (equivalent command for xselection)
echo sometext | xsel -b                 # To set CLIPBOARD buffer
echo sometext | xselection CLIPBOARD -  # (equivalent command for xselection)
xselection CLIPBOARD sometext           # (equivalent command for xselection)

xvkbd

xvkbd is a virtual (graphical) keyboard program for X Window System which provides facility to enter characters onto other clients (softwares) by clicking on a keyboard displayed on the screen. It can also be used to send keyboard events through command-line:

xvkbd -xsendevent -text text-string

xwininfo

xwininfo is the window information utility for X. Could be used to find the geometry of a window. See here

xwininfo                               # Return information on the clicked windows
xwininfo -root                         # ... or on the root window
xwininfo -root |awk '/Width/{print $2} # Get Width of the current desktop

Text Manipulation

column

Format input into multiple columns

cat | column -l
123 456
98765 98765
^D
# 123   456
# 98765 98765

csvquote

csvquote is a nice wrapper tool to pre-/post-process CSV files to ease processing by tools like AWK (to handle comma within quotes)

csvquote foobar.csv | cut -d ',' -f 5 | sort | uniq -c | csvquote -u
cat foobar.csv | csvquote | cut -d ',' -f 7,4,2 | csvquote -u
csvquote -t foobar.tsv | wc -l
csvquote -q "'" foobar.csv | sort -t, -k3 | csvquote -u
csvquote foobar.csv | awk -F, '{sum+=$3} END {print sum}'

cut

TBC

paste

TBC

fold

fold -w 40 FILE              # Wrap FILE to 40-char width

expand

expand -t 4 FILE             # Convert tabs to spaces (4)
expand -t 4,10 FILE          # Set explicit tab positions

fmt

TBC

less

command | less -S                      # To wrap (aka fold) long-lines (or use '-','S' within less)

nl

nl numbers lines of files

nl -ba FILE                  # Number *all* lines of FILE

rl, shuf

rl is a command-line tool that reads lines from an input file or stdin, randomizes the lines and outputs a specified number of lines (requires package randomize-lines).

shuf generates random permutations.

They are alternative to sort -R (note that the latter does not work with duplicate lines).

sed

Moved to page dedicated to Sed.

sort

sort is the well-know text sort utility.

sort -R shuffle.txt                           # shuffle the lines (use random hash) - fail with duplicate lines
nl -ba file | sort -R | sed 's/^.*[0-9]\t//'  # ... idem, but workaround for dup lines

sort by default takes into account the current locale. To get the traditional sort order that uses native byte values:

LC_ALL=C sort myfile.txt                      # traditional sort, indep of current locale

Use -o file to sort in-place, or use the sponge utility:

sort file -o file                             # sort in-place
sort file | sponge file                       # soak sorted file and write it back

Use LC_ALL=C for huge speed improvement. Also increasing the internal buffer helps:

LC_ALL=C sort -S 2G file

tr

tr translates or deletes characters.

Some basic uses:

echo lowercase | tr a-z A-Z                # turns lowercase into uppercase
echo lowercase | tr [:lower:] [:upper:]    # idem
echo encrypt me please | tr a-z n-za-m     # a very simple rot13 implementation

A powerful rot-n implementation [20]:

dual=abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
phrase='hello there'
n=13
newphrase=$(echo $phrase | tr ${dual:0:26} ${dual:$n:26})
echo ${newphrase}

tr can also be used to split words into newlines:

tr ' ' '\n' file

figlet, toilet... (fun)

Some funny banner-like tool:

• figlet [21]
• toilet [22]
• cowsay
• ponysay
• lolcat

Image manipulation

netpbm

Netpbm is a package of graphics programs and programming libraries. There are over 220 separate programs in the package, most of which have pbm, pgm, ppm, or pnm in their names. For example, pnmscale and giftopnm.

For example, you might use pnmscale to shrink an image by 10%. Or use pnmcomp to overlay one image on top of another. Or use pbmtext to create an image of text. Or reduce the number of colors in an image with pnmquant.

anytopnm myfile.gif | pnmtopng > myfile.png                    # Convert a GIF file to PNG
anytopnm oll28.gif | pnmflip -r90 | ppmtogif > oll28-90.gif    # Rotate and output GIF

JPEG file manipulation can be done with djpeg and cjpeg from package libjpeg-turbo-progs:

djpeg foo.jpg | pnmflip -r90 | cjpeg)    # Lossy rotate of jpeg

Development

addr2line

addr2Line converts addresses into file names and line numbers.

ar

ar creates, modifies, and extracts from archives. It can be used to generate static libraries (.a files) for use with gcc.

diff

Use diff to compare 2 files/directories together and/or to generate patch files. Use colordiff (package colordiff) to have colored output.!

diff    old new                        # View difference (default ed script format)
diff -u old new >new.patch             # Generate a patch file (universal format)
diff -y old new                        # View difference side-by-side (as with sdiff)

#Color with DIFF:
colordiff -u old new                   # colordiff can also colorize a diff output package colordiff
diff old new | colordiff               # ... idem

#Color with WDIFF:
wdiff -n old new | colordiff           # But need option '-n'

#Piping to LESS with COLORS
diff -u old new | colordiff | less -R  # To keep color with pager

#DIFF / WDIFF combo
diff --strip-trailing-cr -bu old new | wdiff -d | colordiff | less -R             # Handy when wdiff miss some options...

Patch can be applied with the command patch (see below). Be careful regarding file names / directory structures when generating patch files! Sometimes you'll need to edit the generated patch files to fix the directory structure.

ldd

ldd print shared object dependencies.

See also readelf and lddtree (package pax-utils).

ldd /bin/ls
#       linux-vdso.so.1 (0x00007ffcc3563000)
#       libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f87e5459000)
#       libcap.so.2 => /lib64/libcap.so.2 (0x00007f87e5254000)
#       libc.so.6 => /lib64/libc.so.6 (0x00007f87e4e92000)
#       libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f87e4c22000)
#       libdl.so.2 => /lib64/libdl.so.2 (0x00007f87e4a1e000)
#       /lib64/ld-linux-x86-64.so.2 (0x00005574bf12e000)
#       libattr.so.1 => /lib64/libattr.so.1 (0x00007f87e4817000)
#       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f87e45fa000)

lddtree

lddtree

readelf

readelf displays information on ELF files.

# View libraries linked in .elf (but only directly linked, these may yet load other libs)
readelf -d obs-ios-camera-source.so | grep NEEDED
#  0x0000000000000001 (NEEDED)             Shared library: [libobs.so.0]
#  0x0000000000000001 (NEEDED)             Shared library: [libavcodec.so.58]
#  ...

patch

Use patch to apply a patch previously generated with the command diff. patch use is pretty much straightforward, one mainly has to pay attention to directory structure / blanks.

Complete example using diff:

$ cp project project-patched     
$ cd project-patched
$ vi somefile                                             # We start modifying the copy
$ cd ..
$ diff -u project project-patched >project.patch          # We generate the patch file (universal format)
$ cd project                                              # We cd into project directory to patch
$ patch -lNp1<../project.patch                            # We apply the patch in place

• -l — ignore white spaces (very handy when copying patch from internet page for instance).
• -N — ignore patch that seems reversed or already applied (same as --forward).
• -p1 — tells patch to ignore the first (i.e. one level of) directory in the file name given in the patch header.

The value to use for option p depends actually on the patch header:

• First example - same root directory for both to and from file

--- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2/jpg.c       2009-08-25 16:06:05.242378300 +0200

$ patch -lNp0 <project.patch                        # First example

• second example - different root directory between to and from file

--- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2-patched/jpg.c       2009-08-25 16:06:05.242378300 +0200

$ cd project_dir; patch -lNp1 <../project.patch     # Second example

vbindiff

vbindiff — hexadecimal file display and comparison

vbindiff file1.bin file2.bin               # Compare and view 2 binary files

Scripting

Some commands very useful when writing scripts

command

From the manpage, command execute a simple command...

GDB=$(command -v gdb)                  # Get path to gdb

command -v behaves much like which, except that on Bash:

• command is a shell built-in.
• it treats aliases differently.

Miscellaneous

base64

base64 from coreutils

echo QWxhZGRpbjpvcGVuIHNlc2FtZQ== | base64 --decode
echo `echo QWxhZGRpbjpvcGVuIHNlc2FtZQ== | base64 --decode`

cksum / jacksum / md5sum / sha1sum / sum

cksum file                                              # Generate a CRC similar to CRC32, but in decimal, and with length appended.
cksum -o3 file                                          # Generate a CRC32 checksum (BSD only)
sum file                                                # Generate a checksum
md5sum file                                             # Hash file using MD5
sha1sum file                                            # Hash file using SHA1
jacksum -a crc32 file                                   # Hash using CRC32 (dec)
jacksum -a crc32 -E hexup file                          # Hash using CRC32 (hex)

date

date +'%F %T'                # YYYY-mm-dd HH:MM:SS
date +'%Y%m%d%H%M%S"         # YYYmmddHHMMSS
date +'%s'                   # Get current epoch time (seconds since 1970-01-01 00:00:00 UTC)
date --date='@1498485311'    # Convert epoch to human readable time

direnv

direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.

• Alternatives: https://github.com/gyf304/dotenv

# Create a new folder for demo purposes.
$ mkdir ~/my-project
$ cd ~/my-project

# Show that the FOO environment variable is not loaded.
$ echo ${FOO-nope}
nope

# Create a new .envrc. This file is bash code that is going to be loaded by
# direnv.
$ echo export FOO=foo > .envrc
.envrc is not allowed

# The security mechanism didn't allow to load the .envrc. Since we trust it,
# let's allow its execution.
$ direnv allow .
direnv: reloading
direnv: loading .envrc
direnv export: +FOO

# Show that the FOO environment variable is loaded.
$ echo ${FOO-nope}
foo

# Exit the project
$ cd ..
direnv: unloading

# And now FOO is unset again
$ echo ${FOO-nope}
nope

dog

dog is better than cat. It writes the contents of each give file, URL, or the standard input to standard output.

dtach

dtach simple program that emulates the detach feature of screen. A session in dtach is a single instance in which a program is running under the control of dtach. The program is disassociated from the original terminal, and is thus protected from your original terminal being disconnected for some reason.

echo

echo -e "Some text\n...on 2 lines..."                    # Enable interpretation of backslash escapes (must be quoted!)

expect

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.

• See SMTP#Expect script for testing SMTP

hyperfine

hyperfine is a command-line benchmarking tool (benchmark tool).

hyperfine --warmup 3 'fd -HI "[0-9]\.jpg"' 'find -name "*[0-9].jpg"'

Hyperfine may produce results in CSV, JSON, Markdown.

iconv

locale encoding conversions

konwert

fancy encoding conversions

mc (Midnight Commander)

mc or Midnight Commander is a powerful file manager working in a shell terminal.
See dedicated page on this wiki.

mimencode

binary file conversion for the mail.

munpack

Use munpack to extract MIME attachment from email [23]:

cat Message
# ...
# 
# --PGP_Universal_31BF743E_DCBDB54C_D4397473_15481C77
# Content-Type: application/octet-stream;
# 	name="tuleap_ex.txt.gpg"
# Content-Transfer-Encoding: BASE64
# Content-Disposition: attachment;
# 	filename="tuleap_ex.txt.gpg"
# 
# hQIOA1uCgmTNqsbUEAgAqbp/CXJfNQmZu8RHtZaP6Fm5gi+sjyothO19faktgOFrVTsame5ZW8yi
# ...
# ...
# NnRRvRI+TcohkYsisQy4oiUA
# --PGP_Universal_31BF743E_DCBDB54C_D4397473_15481C77--
# 
sudo apt install mpack
munpack -f Message

mrxvt

A multi-tab version of rxvt (terminal-emulator).

parallel

parallel runs many job in parallel.

parallel -i chronic sh -c "date; echo {}"

Note that this is parallel from moreutils. There is also a separate package parallel with different options.

recode

recode /cl../cr <dos.txt >mac.txt
recode /cr.. <mac.txt >unix.txt
recode ../cl <unix.txt >dos.txt
recode charset1/surface1..charset2/surface2 <input.txt >output.txt
recode /QP.. <qp.txt >normal.txt                                    # To convert quoted-printable text

charset		surface
us	ASCII (7 bits)	/cr	Carriage return as end of line (Mac text)
l1	ISO Latin-1 (ISO-8859-1, Western Europe, 8 bits)	/cl	Carriage return line feed as end of line (DOS text)
EUCJP	EUC-JP for Japanese (Unix)	/	Line feed as end of line (Unix text)
SJIS	Shift-JIS for Japanese (Microsoft)	/d1	Human readable bytewise decimal dump
ISO2022JP	Mail encoding for Japanese (7 bits)	/x1	Human readable bytewise hexidecimal dump
u2	UCS-2 (Universal Character Set, 2 bytes)	/64	Base64 encoded text (see also base64) from coreutils.
u8	UTF-8 (Universal Transformation Format, 8 bits)	/QP	Quoted-Printable encoded text

reformime

MIME E-mail reformatting tool

screen

Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.

See also tmux for a better alternatives that supports splitting in panes, mousewheel support...

Launch with

screen                                      # Start a new screen session
screen -r                                   # Resumes a detacked screen session
screen -d -r                                # Reattach a session and if necessary detach it first
screen -d -R                                # Reattach a session and if necessary detach or even create it first
screen -D -r                                # Reattach a session. If necessary detach and logout remotely first
screen -D -R                                # Attach here and now, ie:
                                            # - If no session is running, create it
                                            # - Otherwise reattach the session. If necessary detach/logout remotely first

For convenience one may add the following alias to .bashrc:

alias screen=screen -D -R

Useful shortcuts:

key	description
C-a c	Open a new instance
C-a Space	Switch to next instance
C-a d	Detach from screen. Reattach later with screen -r
C-a D D	Power detach, i.e. detach and logout

byobu is a wrapper around screen, which makes it prettier and more powerful (see this also).

Use reptyr to capture an already running process and attach it to a new terminal. Very handy if you start a process, and you regret not having started screen first...

sponge

sponge (package moreutils) soaks up standard input and write to a file. It can be used to easily edit file in-place:

sed -r '...' FILE | grep ... | sponge FILE                   # Pipeline from and to same file!

tabs

Set tabulation size of the terminal.

tabs 4               # Set default tab size to 4
tabs 8               # Set default tab size to 8 (the default)

Note that some applications are not impacted by this setting or have alternate solution:

less -x1,5                                       # LESS - sets tabs
git config --global core.pager 'less -x1,5'      # GIT
diff -t --tabsize=4                              # DIFF - alternate solution

tesseract

A tool to perform OCR.

Example of script AskUbuntu:

#!/bin/bash

SCR_IMG=".screentemp.png"
TEMP_TXT=".screentext.txt"

gnome-screenshot -a --file=$SCR_IMG
tesseract $SCR_IMG $TEMP_TXT -l eng
cat $TEMP_TXT* | xsel -b

rm $SCR_IMG $TEMP_TXT*

timeout

timeout runs a command with a time limit.

On MinGW, timeout is not available. Instead there is a bash-script equivalent.

tmux

See page Tmux.

uuencode, uudecode

Binary file conversion for Unix.

wc

Counts lines, words and character in a file

xargs

For more examples, see combinations with the command find

#By default, xargs cuts at WHITE SPACE and NEWLINE. Use -d '\n' to only cut at NEWLINES
for i in $(locate .pdf); do basename $i; done      # WRONG
locate .pdf | xargs -rd '\n' -n 1 basemane          # Correct

Hex Tools

hd / hexdump

ASCII, decimal, hexadecimal, octal dump.

hexdump -e '"%2x"' <myfile>             # Convert myfile into a long hexadecimal string - ! See DOUBLE-QUOTED parameter
hexdump -C <myfile>                     # Canonical hex + ascii display (16 bytes)
hd <myfile>                             # (idem)

See also od and xxd. There are also hex editors, like hexer and hexedit (adapts to width of the terminal!).

od

od - dump files in octal and other formats

od -xc file                                              # Output file in ascii and hex output (2-byte, ! little-endian)
od -t x1 -a --width=32 testvar.sh                        # Output 32-byte at once

See also hd / hexdump and xxd.

xxd

Make a hexdump or do the reverse. This tool is delivered along with vim. Probably the BEST out there

xxd -g8 -c32 <file>                        # Output 32 bytes, grouped in 8-byte columns
xxd -p -c64 <file>                         # Output 64 bytes per line, plain (postscript) mode
echo 202122 | xxd -r -p                    # Convert hexdump to a binary string
echo -n ' !"' | xxd -p                     # Convert binary string to hexdump - DON'T FORGET -n
echo -n ' !"' | xxd -u -p                  # Convert binary string to hexdump - DON'T FORGET -n (uppercase)

See also hd / hexdump and od.

xxd can be easily used in Vim to hexedit a file:

" To convert current buffer to hex
:%xxd
" To convert current buffer from hex
:%xxd -r

Security

openssl rand

openssl rand generates random strings:

openssl rand -hex 16                      # Generate a 16-byte random string in hexadecimal
openssl rand -base64 16                   # Generate a 16-byte random string in base64

Desktop

notify-send

notify-send is a program to send desktop notifications.

notify-send "summary" "body message"

If messages are not visible, use either -t 0 to remove time limit, or critical urgency [24]:

 
notify-send -u critical "Urgent" "This is urgent!"
notify-send -t 0 "Nag" "This is a nag screen."

notify-send uses icons from /usr/share/icons/gnome/32x32 and /usr/share/notify-osd/icons/.... Any file in any sub-folder can be used directly, or a specific image file can be given [25]:

notify-send -i face-laugh "I am happy"
notify-send -i task-due "Time to work dude!"
notify-send -i ~/myicon.png "This is my icon"

Multimedia

paplay

paplay is the best alternative on "modern" linux

paplay /usr/share/games/xboard/sounds/woodthunk.wav

play

Play sound with ALSA. But often get under-run warnings (see Linux audio). Better use paplay.

play /usr/share/games/xboard/sounds/woodthunk.wav
play --single-threaded /usr/share/games/xboard/sounds/woodthunk.wav    # Does not help

Python modules

Reference: https://til.simonwillison.net/python/stdlib-cli-tools

HTTP server

python -m http.server

Base64

python3.11 -m base64 -h

JSON

echo '{"foo": "bar", "baz": [1, 2, 3]}' | python -m json.tool

Calendar

python -m calendar