Linux Commands: Difference between revisions

From miki
Jump to navigation Jump to search
(→‎sed: Using insert/append text commands i\ a\)
 
(284 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== dd ==
== References ==
From '''The Linux Documentation Project''':
* [http://tldp.org/LDP/abs/html/textproc.html 16.4. Text Processing Commands]

== File System ==

=== [http://betterthangrep.com/ ack] ===
'''ack''' is a grep-like program specifically for large source trees. It is distributed via package '''ack-grep'''. Add a symlink for convenience:
<source lang=bash>
sudo ln -s /usr/bin/ack-grep /usr/local/bin/ack
</source>

Features:
* It's '''blazingly fast''' because it only searches the stuff you want searched.
* ack '''ignores most of the crap''' you don't want to search
* '''Lets you specify file types''' to search, as in --perl or --nohtml.
* '''Color highlighting''' of search results.
* Uses real '''Perl regular expressions''', not a GNU subset.

<source lang="bash">
# grep pattern $(find . -type f | grep -v '\.svn')
ack pattern # Ack ignore CVS, SVN dir
# grep pattern $(find . -name '*.pl' -or -name '*.pm' -or -name '*.pod' | grep -v .svn)
ack --perl pattern # Specify file type to search
ack -f --perl > all-perl-files # Create rapidly list of files
</source>

Supported file types (more [http://betterthangrep.com/ here]):
<source lang="text">
--[no]asm .asm .s
--[no]batch .bat .cmd
--[no]binary Binary files, as defined by Perl's -B op (default: off)
--[no]cc .c .h .xs
--[no]cpp .cpp .cc .cxx .m .hpp .hh .h .hxx
--[no]css .css
--[no]hh .h
--[no]html .htm .html .shtml .xhtml
--[no]java .java .properties
--[no]js .js
--[no]make Makefiles (including *.mk and *.mak)
--[no]perl .pl .pm .pod .t
--[no]php .php .phpt .php3 .php4 .php5 .phtml
--[no]python .py
--[no]ruby .rb .rhtml .rjs .rxml .erb .rake .spec
--[no]shell .sh .bash .csh .tcsh .ksh .zsh
--[no]skipped Files, but not directories, normally skipped by ack (default: off)
--[no]text Text files, as defined by Perl's -T op (default: off)
--[no]vim .vim
--[no]xml .xml .dtd .xsl .xslt .ent
</source>

=== [http://geoff.greer.fm/2011/12/27/the-silver-searcher-better-than-ack/ ag] ===
'''ag''' ([http://geoff.greer.fm/2011/12/27/the-silver-searcher-better-than-ack/ home], [https://github.com/ggreer/the_silver_searcher github], [https://github.com/mileszs/ack.vim ack.vim], [https://github.com/epmatsw/ag.vim ag.vim]) is the ''silver searcher'', and is actually much better than ''ack'', which in turn is much better than ''grep'' ;-). '''Update''': turns out that [https://github.com/BurntSushi/ripgrep ripgrep] is an even faster alternative.

On recent distribution, install from package:
<source lang=bash>
sudo apt install silversearcher-ag
</source>

To install from [https://github.com/ggreer/the_silver_searcher.git git] (see {{file|README.md}}):
<source lang=bash>
sudo apt-get install software-properties-common
sudo apt-get install -y automake pkg-config libpcre3-dev zlib1g-dev liblzma-dev
git clone https://github.com/ggreer/the_silver_searcher.git
cd the_silver_searcher/
./build.sh
sudo make install
</source>

;Tips
* Use <code>/b</code> to find ''whole word'' matches only, or better <code>-w</code>:
<source lang=bash>
ag "\bWHOLEWORD\b"
ag -w WHOLEWORD
</source>

=== chattr ===
'''chattr''' change file attributes on a Linux file system.

See <code>man chattr</code> for a list of attributes (immutable...)

=== cp ===

To copy '''content''' of a directory into another:
<source lang="bash">
cp -a /source/. /dest/
</source>
This copy recursively and preserve attributes <code>-a</code>, and copy the '''content''' with special syntax <code>source/.</code> with trailing dot.

To copy a directory recursively, create hard links instead of new copies (but creates new directories, not links):
<source lang="bash">
cp -lr <src> <target>
</source>

See [[#cpg]] for a patch that adds a progress bar.

=== cpg ===

<code>cpg</code> is a patched version of [[cp]] that adds a progress bar, [https://github.com/jarun/advcpmv advcpmv(github)].

<source lang="python">
cpg -ag src1 dst2 # Copy recursively with progress
</source>

Alternatives are to use <code>rsync --progress</code> (or <code>rsync -P</code>), or [[#gcp]].

=== dd ===
Convert and copy a file.
Convert and copy a file.


Line 6: Line 112:
This will force '''dd''' to update its status every 5sec.
This will force '''dd''' to update its status every 5sec.


'''Transfer speed is highly dependent on the block size!''' So make sure to use <code>-bs=16M</code> or something like that:
== dpkg ==
dd if=/dev/sda1 of=backup-sda1 bs=16M
Package manager for Debian / Ubuntu. See [[Package Management#Ubuntu / Debian]]


Other recommended values for <code>-bs</code> are: <code>1MB</code> [http://unix.stackexchange.com/questions/9432/is-there-a-way-to-determine-the-optimal-value-for-the-bs-parameter-to-dd/], <code>64k</code> [http://superuser.com/questions/234199/good-block-size-for-disk-cloning-with-diskdump-dd] [http://www.mail-archive.com/eug-lug@efn.org/msg12073.html].
== echo ==

Note: to copy a whole file or device <code>cat</code> or <code>pv</code> are better alternatives (see [[Linux Disk Management]]). <code>cat</code> is at least faster than <code>dd</code> with default parameters [http://unix.stackexchange.com/questions/9432/is-there-a-way-to-determine-the-optimal-value-for-the-bs-parameter-to-dd/9492#9492], and <code>pv</code> looks for optimal block size to get highest speed.
<source lang=bash>
dd if=file1 of=file2 # slow
cat file1 > file2 # fast
dd if=file1 of=file2 bs=16M # faster (maybe)
pv file1 > file2 # fastest
</source>

Interesting links:
* '''[https://github.com/sampablokuper/dd-opt dd-opt]''', an utility to find optimal value for <code>bs</code> parameter.
* A complete blog post about [http://blog.tdg5.com/tuning-dd-block-size/ Tuning dd block size]
* [http://unix.stackexchange.com/questions/9432/is-there-a-way-to-determine-the-optimal-value-for-the-bs-parameter-to-dd/ Discussions on stackexchange.com], also suggesting that <code>cat</code> could be used too.
* ... but [http://unix.stackexchange.com/questions/12532/dd-vs-cat-is-dd-still-relevant-these-days/12538#12538 dd still has many other uses that cat does not provide].
* ... for instance, create sparse files, and mount it as filesystem loopback:
<source lang=bash>
# Create sparse file
dd of=sparse-file bs=1 count=0 seek=10 # SAME AS: truncate -s 10GB sparse-file
# Mount it
loop=`losetup --show -f sparse-file`
mkfs.ext4 $loop
mkdir myloop
mount $loop myloop
</source>
* [https://www.vidarholen.net/contents/blog/?p=479 Useless Use Of dd]
<source lang="bash">
<source lang="bash">
# Write myfile.iso to a USB drive
echo -e "Some text\n...on 2 lines..." # Enable interpretation of backslash escapes (must be quoted!)
# !!! with -a, cp may try to create a new block device!
cp myfile.iso /dev/sdb

# Rip a cdrom to a .iso file
cat /dev/cdrom > myfile.iso

# Create a gzipped image
# !!! w/o redirection, gzip may skip non-regular file /dev/sdb
gzip -9 < /dev/sdb > /tmp/myimage.gz
</source>
</source>


== find ==
=== entr ===
'''entr''' runs arbitrary commands when files change.

For instance:
<source lang=bash>
find -name *.c | entr make
</source>

=== fatsort ===
{{deb|fatsort}} sorts directory structures of FAT filesystems. Many MP3 hardware players don't sort files automatically but play them in the order they were transmitted to the device. FATSort can help here.

<source lang=bash>
sudo fatsort /dev/sdb1
</source>

=== fd ===
'''[https://github.com/sharkdp/fd fd]''' is a simple, fast and user-friendly alternative to 'find'.

<source lang="bash">
<source lang="bash">
fd
find . -exec echo '{}' \; #semi-colon must be escaped!
fd app
find . -exec echo one '{}' \; -exec echo two '{}' \; #to execute several commands on one match
fd sh
fd sh --type f
fd -e md
fd -e md --exec wc -l
fd --hidden sample
</source>
</source>


== grep ==
=== find ===
'''Tip:''' Consider using <code>fd</code> as simpler and faster alternative.

See [http://www.kalamazoolinux.org/tech/find.html here] for further examples on how to combine '''find''' and '''xargs'''.

;{{red|FREQUENT PITFALLS!}}
* '''{{red|ESCAPE THE SEMI-COLON}}''' &mdash; The semi-colon <tt>;</tt> must be escaped in Bash !
* '''{{red|1=USE <code>-</code> OR <code>+</code> WITH NUMERICAL VALUES}}''' &mdash; for instance <code>-mtime 1</code> means ''<u>exactly</u> one day ago'', whereas <code>-mtime -1</code> means ''<u>within</u> the last day''.

<source lang="bash">
find . -name "*.jpg" # find files/directories whose name matches specified pattern
find . -path "*/cur/*" # find files/directories whose path spec matches specified pattern
find . -type d -name "cur" # find only directories whose name matches specified pattern
find . -exec echo '{}' \; # semi-colon must be escaped!
find . -exec echo one '{}' \; -exec echo two '{}' \; # to execute several commands on one match
find . -name 'dirname' -prune -o -iname '*.txt' -print
# Find all text files, but exclude any directory named dirname
find . -name 'dirname' -prune -false -o -iname '*.txt' # ... idem (using -false, but no longer -print)
find . -path './path' -prune -false -o -iname '*.txt' # Find all text files, but exclude path ./path
find . -iname '*.zip' -exec sh -c 'unzip -l "$0" | egrep -o " [0-9]+ file"' '{}' \;
# How to use pipe in -exec.
find www -name '*.pdf' -printf %P\\n # Print PDF files in www/ but remove the leading www/ in output (%P)
</source>

'''Accelerating''' find:
<source lang="bash">
find / -mount -name "*splash*" # Only search on CURRENT file system (same as find -xdev)
</source>

Use also '''xargs''' to accelerate find:
<source lang="bash">
find . -name \*.jpg | xargs -r echo # much faster than find -exec
find . -name \*.jpg | xargs -rn 2 echo # limit #args passed to echo to 2 max.
find . -name \*.jpg -print0 | xargs -r0 ls # Use NULL separator if files contains space or special char !!!

# find -exec command \; and xargs -i are equivalent
find . -name "*.txt" -exec echo '{}' is found. \; # ONE at a time, but {} can be anywhere - and MULTIPLE times
find . -name "*.txt" -print0 | xargs -r0 -i echo '{}' is found. # ... same, with xargs
# find -exec command {} + and xargs are equivalent
find . -name "*.txt" -exec echo The files are '{}' + # FASTER - much like xargs. Only append AT THE END!
find . -name "*.txt" -print0 | xargs -r0 echo The files are # ... same with xargs

# With mv, use -t DIRECTORY to give destination directory first
find . -size +100M -print0 | xargs -0 mv -t bigfiles/ # -t DIR inverts usual SOURCE... DESTINATION order
find . -size +100M -exec mv -t bigfiles/ {} + # ... idem, without xargs
</source>

=== [http://www.pixelbeat.org/fslint/ FSlint] ===
'''FSlint''' is a utility to find and clean various forms of lint on a filesystem. I.E. unwanted or problematic cruft in your files or file names. For example, one form of lint it finds is '''duplicate files'''.

=== gcp ===
A variant of [[cp]], but with progress bar:

<source lang="bash">
gcp -pR dir1 .
</source>

If complains about missing dbus:

<source lang="bash">
dbus-launch gcp -pR dir1 .
</source>

An alternative is to use a patched version of [[cp]].

=== getfattr / setfattr ===
'''getfattr''' and '''setfattr''' gets/sets extended attributes of filesystem objects.

{{red|!!! These tools are really misleading, and many attributes are hidden. Usually you need to list the attributes explicitly !!!}}

<source lang=bash>
getfattr -d file # This SUPPOSEDLY returns *ALL* attributes, but actually *MANY* missing
getfattr -m - file # This MIGHT return more attributes
getfattr -n system.ntfs_attrib file # View ntfs-3g special attributes
getfattr -d -e hex file # View attribute values in HEX
</source>

=== grep / zgrep ===
<source lang="bash">
<source lang="bash">
grep -Rsl PATTERN [FILE] # Recursive, no error output, only list filename
grep -Rsl PATTERN [FILE] # Recursive, no error output, only list filename
Line 31: Line 270:
grep -R -include=*.in PATTERN * # Search recursively through folders, limiting to files matching pattern "*.in"
grep -R -include=*.in PATTERN * # Search recursively through folders, limiting to files matching pattern "*.in"
grep -R PATTERN *.in # Idem, but matching pattern "*.in" also applies to folders.
grep -R PATTERN *.in # Idem, but matching pattern "*.in" also applies to folders.
grep -o PATTERN [FILE] # (--only-matching) Print only the matched (non-empty) parts of a matching line.
</source>
</source>


'''zgrep''' works like ''grep'', but also accepts compressed (.gz) files.
== [http://netcat.sourceforge.net/ netcat] ==

<source lang="bash">
zgrep PATTERN *.txt *.txt.gz # Search in .txt file (compressed or not)
</source>

=== gzip / gunzip / zcat ===
<source lang="bash">
gzip -c FILE # Compress FILE to stdout
gunzip -c FILE # Uncompress FiLe to stdout
zcat FILE # Equivalent to gunzip -c FILE
</source>

=== ls ===
Listing directory content:
<source lang="bash">
ls --full-time # To list files full date & time
ls -d directory-name # List directory entries, not content
# eg. ls -d /etc/rc*
ls -lS | head # List biggest files first - print only first 10 entries
ls -lt | head -n 20 # List most recent files first, print only 20 first entries
</source>

=== lsattr ===
'''lsattr''' list file attributes on a Linux second extended file system.

<source lang=bash>
sudo lsattr
</source>
---------------- ./memtest86+.elf
---------------- ./System.map-3.13.0-39-generic
---------------- ./vmlinuz-3.13.0-39-generic
---------------- ./initrd.img-3.13.0-39-generic
----------I----- ./grub
---------------- ./abi-3.13.0-39-generic
---------------- ./memtest86+_multiboot.bin
---------------- ./memtest86+.bin
---------------- ./config-3.13.0-39-generic
---------------- ./lost+found

=== ntfsundelete ===
'''ntfsundelete''' recover a delete file from an NTFS volume. Does not scan the disk.

See [http://linuxpoison.blogspot.be/2010/10/recover-deleted-files-from-ntfs.html], [http://www.howtogeek.com/howto/13706/recover-deleted-files-on-an-ntfs-hard-drive-from-a-ubuntu-live-cd/].

=== par2 ===
{{deb|par2}} is a very useful program for creating and using PAR2 files to detect damage in data files and repair them if necessary.

From the manpage, using an example 800MB {{file|test.mpg}}:
<source lang=bash>
par2 create test.mpg # Will create *.par2 files, for a total of roughly 40MB (5% redundancy)
par2 verify test.mpg.par2 # to verify integrity
par2 repair test.mpg.par2 # to repair
</source>

{{file|*.par2}} files stores error correction informations, in a number of recovery blocks. Repair is possible as long as there are enough recovery blocks available (on Usenet era, it was useful to have {{file|*.par2}} files of increasing size such that to allow users to minimize bandwidth usage).

par2 is particularly useful to protect MP3 files on USB sticks (that have frequent FS failure over time):
<source lang=bash>
par2 create check.par2 *.mp3 # Generate recovery blocks (5% redundancy)
par2 verify check.par2 # Verify
par2 repair check.par2 # Repair
</source>

To increase redundancy:
<source lang=bash>
par2 create -r10 check.par2 *.mp3 # Generate recovery blocks (10% redundancy)
</source>

Note: to simulate file corruption, we can use:
<source lang=bash>
# Simulate 25kB corruption, from 10k-th byte.
dd if=/dev/zero of=output.mp3 bs=1024 count=25 seek=10 conv=notrunc
</source>

=== progress ===
'''progress''' [https://www.tecmint.com/progress-monitor-check-progress-of-linux-commands/] watches progress of other utilities like ''cp'', ''mv'', ''dd'', ''tar''.

<source lang=bash>
# display estimated I/O throughput and estimated remaining time for on going coreutils commands
progress -w

# Start an heavy command and monitor it with -m and $!
tar czf images.tar.gz linuxmint-18-cinnamon-64bit.iso CentOS-7.0-1406-x86_64-DVD.iso CubLinux-1.0RC-amd64.iso | progress -m $!
</source>

=== pv ===

'''pv''' monitors the progress of data through a pipe. It can also be used to copy files or entire volume very rapidly.

For instance:
<source lang=bash>
pv </dev/sda >/dev/sdb
</source>

Also over the network using <code>nc</code>:
<source lang=bash>
# On Host A (receiver):
nc -l 2222 > /dev/sda
# On Host B (sender):
pv </dev/sda | nc hosta 2222 # 111MB avg throughput on direct link (cross-cable), SSD hard disks
</source>

=== rdfind ===
'''rdfind''' finds duplicate files, very rapidly.

<source lang="bash">
rdfind /foo/dir /bar/dir # Find duplicates in give dir and tell what could be gained
rdfind -makehardlinks true /foo/dir /bar/dir # Remove dups by creating hardlinks
</source>

=== rg ===
'''rg''' or '''ripgrep''' is an extremelly fast alternative to ''grep'' (in fact even faster than ''ag'' the silver-searcher).

See [https://github.com/BurntSushi/ripgrep ripgrep on GitHub] for {{deb|ripgrep.deb}} package.

=== rsync ===
Perfect copy over network (see [[Linux_Disk_Management#rsync|this page]] for details)

<source lang=bash>
#If needed, pre-activate sudo on remote system. Flag -t required to solve 'sudo: no tty present and no askpass program specified'
#
# Also, this requires the following line in /etc/sudoers:
#
# Defaults !tty_tickets
#
stty -echo; ssh -t user@server sudo -v; stty echo

sudo rsync -aHAXS --delete --rsync-path "sudo rsync" --numeric-ids -h -v --exclude='lost+found' user@server:/remote/path /local/path
</source>

More basic use of rsync:
<source lang="bash">
sudo rsync -aP src1 dst2 # Copy with progress
</source>

=== scalpel ===
'''scalpel''' recover files using a header / footer database (see [http://linuxpoison.blogspot.be/2012/05/recover-deleted-files-using-scalpel.html]).

Install ''scalpel'', then edit <tt>/etc/scalpel/scalpel.conf</tt> to select which file types to recover (based on header/footer).

<source lang=bash>
sudo apt-get install scalpel
sudo vi /etc/scalpel/scalpel.conf
</source>

Here we enable ''jpg'' and ''mp3'' (see [http://fossies.org/linux/misc/scalpel-2.0.tar.gz:a/scalpel-2.0/scalpel.conf] for more):
<source lang=text>
# GIF and JPG files (very common)
# gif y 5000000 \x47\x49\x46\x38\x37\x61 \x00\x3b
# gif y 5000000 \x47\x49\x46\x38\x39\x61 \x00\x00\x3b
jpg y 200000000 \xff\xd8\xff\xe0\x00\x10 \xff\xd9
jpg y 200000000 \xff\xd8\xff\xe1 \xff\xd9
# MP3
mp3 y 8000000 \xFF\xFB??\x44\x00\x00
mp3 y 8000000 \x57\x41\x56\45 \x00\x00\xFF\
mp3 y 8000000 \xFF\xFB\xD0\ \xD1\x35\x51\xCC\
mp3 y 8000000 \x49\x44\x33\
mp3 y 8000000 \x4C\x41\x4D\x45\
</source>

Start scalpel:
<source lang=bash>
sudo scalpel /dev/sda1 -o output_dir # output_dir must *not* be on disk being scanned
</source>


=== tar ===
<tt>tar</tt> archives and restores timestamps. When run as '''root''', <tt>tar</tt> will also restore ownership and permissions.
<source lang="bash">
tar -cvzf archive.tgz somedir/ # Archive & GZIP compress the directory somedir/ (recursively)
tar -cvjf archive.tar.bz2 somedir/ # ... Use BZIP2 insteadl
tar -cvzf archive.tgz --exclude=notme somedir/ # ... exclude any file matching name
tar -cvzf archive.tgz --exclude=somedir/not/me somedir/ # ... exclude file at specific location

tar -xvzf archive.tgz # Extract archive (will create somedir/)
tar -xvzf archive.tgz -C targetdir # ... to specified directory
tar -tvzf archive.tgz # List the content (equiv. of tar --list -vzf ...)

tar cf - --sort=name --owner=root:0 --group=root:0 \
--mtime='UTC 2019-01-01' . | gzip -n > invariant.tgz # Create a stable archive w/o timestamps, etc
# https://stackoverflow.com/questions/32997526/how-to-create-a-tar-file-that-omits-timestamps-for-its-contents
</source>

;Troubleshootigng
* On windows, getting errors like <code>tar: ....myfile.txt: file changed as we read it</code>.
:No fix. We silent the error with the following script:
<source lang=bash>
nice_tar()
{
# Bloody tar returns a warning about modified dir on windows. Must silent the warning.
tar "$@"
RC=$?

[ $RC -eq 0 -o $RC -eq 1 ] && return 0 || return $RC
}
</source>

=== testdisk / photorec ===
'''testdisk''' scans and repairs disk partitions. '''photorec''' recovers lost files from harddisk, digital camera and cdrom (including audio and photo files).

<source lang=bash>
photorec /log /d output_dir /dev/sda1
</source>

=== ug ===
'''ug''' or '''ugrep''' is an extremelly fast alternative to ''grep'' (in fact even faster than ''ag'' the silver-searcher, and comparable speed with '''ripgrep''', but with some more powerful search capabilities).

See [https://github.com/Genivia/ugrep ugrep on GitHub] (also nice instructions for setup on Vim).

=== Miscellaneous ===
;badblocks
:An utility to detect bad sectors on USB flash drive or so.

;dosfslabel
:Change FAT16/FAT32 disk label
<source lang="bash">
sudo dosfslabel /dev/sdc1 UBUNTULIVE
</source>

;mbr
:Install MS-like Master Boot Record (see [http://ubuntuforums.org/archive/index.php/t-339433.html])
<source lang="bash">
sudo apt-get install mbr
install-mbr -i n -p D -t 0 /dev/sda # Fix MBR on /dev/sda
# -i interrupt n = none (do not display a MBR prompt)
# -p partition D = boot the partition with the bootable flag set
# -t timeout 0 = do not wait to boot
dd if=/dev/sda of=opensource.mbr bs=512 count=1
install-mbr -i n -p D -t 0 opensource.mbr # Create a file opensource.mbr containing the generated MBR
</source>

;mkfs
:Format disk utility
<source lang="bash">
sudo mkfs.vfat -F 32 -c /dev/sdc1
</source>

;ms-sys
:Install MS-compatible Master Boot Record (similar to <tt>FIXMBR</tt> on WinXP rescue console, see [http://www.arsgeek.com/2008/01/15/how-to-fix-your-windows-mbr-with-an-ubuntu-livecd/])
<source lang="bash">
# Get ms-sys from http://ms-sys.sourceforge.net/#Download
sudo apt-get install gettext
make
sudo make install
sudo ms-sys -m /dev/sdb
</source>

;relocntfs
:'''relocntfs''' (see [http://trinityhome.org/phpbb2/viewtopic.php?t=622&highlight=relocntfs]) is more or less the equivalent of the '''FIXBOOT''' command in the ''Windows Recovery Console''. It is among other available on the [http://trinityhome.org/Home/index.php?wpid=1&front_id=12 ''Trinity Rescue Kit'']. For instance, to fix the ''boot record'' of a windows partition at <tt>/dev/sda2</tt> issue the commands:
<source lang="bash">
relocntfs /dev/sda2
relocntfs -w /dev/sda2 # To actually write the new boot record
</source>
;stat
:display file or file system status (like file access time, creation time, modification time...)
;[http://mama.indstate.edu/users/ice/tree/ tree]
:List the contents of directories in a tree-like format.

== Network ==

=== autossh ===
See [[SSH Tools#Autossh|Autossh]].

=== dhclient ===
<source lang=bash>
dhclient -r eth1 # Release DHCP lease on eth1
dhclient eth1 # Obtain fresh ip on eth1
</source>

=== [http://linux.about.com/od/commands/l/blcmdl1_dig.htm dig] ===
'''dig''' stands for ''domain information groper''. It replaces the deprecated ''nslookup''. It comes with package '''dnsutils''':
<source lang=bash>
sudo apt-get install dnsutils
</source>

To get ip address of a given host:
<source lang=bash>
dig +short google.com | head -n 1 # In case host has multiple address
</source>

See also <code>getent hosts</code> or <code>hostname</code>.

To do a '''reverse DNS lookup''':
* Use option <code>-x</code> of ''dig'' command:
<source lang=bash>
dig +short -x 82.78.227.176
# 176.176-191.227.78.82.in-addr.arpa.
# ivorde.ro.
</source>

* Or query record <code>PTR</code> in <code>in-addr.arpa</code> domain:
<source lang=bash>
dig +short ptr 176.227.78.82.in-addr.arpa.
# 176.176-191.227.78.82.in-addr.arpa.
# ivorde.ro.
</source>

=== getent ===
Use <code>getent hosts</code> to get ip address of a given host:
<source lang=bash>
getent hosts google.be
getent hosts google.be | awk '{ print $1; exit }' # To get first ip address only
</source>
Note <code>getent</code> may succeed where <code>dig</code> fails (for instance host in {{file|/etc/hosts}} file or hosts with {{file|.local}} suffix).

=== host ===
Use ''host'' to do a '''reverse DNS lookup''', i.e. get hostname from a given IP address:

host 123.45.67.89

''host'' can also do a regular DNS lookup and fetch alias names:

host -t a www.facebook.com
# www.facebook.com is an alias for star.c10r.facebook.com.
# star.c10r.facebook.com has address 173.252.100.27

=== hostname ===
Use <code>hostname -I</code> to get local host ip address:
<source lang=bash>
hostname -I
hostname -I | awk '{ print $1 }' # In case host has multiple interfaces / ip addresses
</source>

=== ifconfig ===
See [[Linux networking]].

=== ip ===
See [[Linux networking]].

=== iwconfig ===
See [[Wifi]].

=== mail ===
'''mail''' &mdash; send and receive mail.

<source lang=bash>
# Using local SMTP
echo my message body here | /usr/bin/mail -s"My subject here" my_email@myserver.org

# Using ISP SMTP - no auth
echo "my message body here" | mail -S 'smtp=smtp.myisp.org:25' -S 'from=first.last@myisp.org' -s 'My subject here' my_email@myserver.org
</source>

=== multitee ===

'''multitee''' sends multiple inputs to multiple outputs.<br/>
Check this [http://code.dogmap.org/fdtools/multitee/ page].<br/>
Original is [http://cr.yp.to/software/multitee-3.0.shar.gz here], or can also be found here on [http://ftp.de.debian.org/debian/pool/main/m/multitee/multitee_3.0.orig.tar.gz Debian].

Here a [https://kiwi.noekeon.org/miki/upload/multitee-3.0-cygwin.patch patch] to build it on ''Cygwin''. The patch ports calls to BSD ''signal.h'' API (<tt>struct sigvec</tt>, <tt>sigvec()</tt>, <tt>sigmask()</tt>, <tt>sigblock()</tt>...) to the POSIX API (<tt>struct sigaction</tt>, <tt>sigaction()</tt>, <tt>sigprocmask()</tt>, <tt>sigfillset()</tt>...):

<source lang="bash">
$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin
</source>

Example of use:
<source lang="bash">
$ multitee 0-1,4,5 4> foo 5> bar # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0 # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1 # same as 'socat - FD:3'
</source>

=== [http://netcat.sourceforge.net/ netcat] or nc===
TCP-IP swiss army knife
TCP-IP swiss army knife
''(equivalent of the telnet program. Check [[wikipedia:netcat]]. Also known as command '''nc''')''.
''(equivalent of the telnet program. Check [[wikipedia:netcat]]. Also known as command '''nc''')''.


See also '''[[Linux Tips#Using Netcat|Using Netcat (Linux Tips)]]'''.
== [http://en.wikipedia.org/wiki/Netstat netstat] ==

Print network connections, routing tables, interface statistics, masqurade connections, and multicast memberships
;Using as telnet-like:
<source lang=bash>
nc www.immie.org 22 # Connect to SSH port
nc www.immie.org ssh # idem, using port name
nc -v www.immie.org ssh # idem, verbose
</source>
Use ''nc'' also to test whether some ports are open, a bit like <code>nmap</code>. For instance:
<source lang=bash>
nc -vz www.immie.org 25 # See if SMTP port is open
# www.immie.org [91.134.134.85] 25 (smtp) open
nc -vz www.immie.org 80 # idem, with http port
</source>

;Using TCP in multi-connect
:Netcat will quit as soon as the first connection terminates. Use option <code>-k</code> to keep the connection alive:
<source lang=bash>
# On DESTINATION host
nc -l -k 6666 # Keep connection alive
# On SOURCE host
echo foo | nc griffin 6666
echo bar | nc griffin 6666
</source>

;Using UDP in multi-connect
:Note that ''When nc is listening to a UDP socket, '''it 'locks on' to the source port and source IP''' of the first packet it receives.'' [http://stackoverflow.com/questions/7696862/strange-behavoiur-of-netcat-with-udp]. So one must give the source port (with option <code>-p</code>) on the SOURCE host as well, or second and next transmissions will be ignored.
<source lang=bash>
# On DESTINATION host
nc -l -u 6666
# On SOURCE host
echo foo | nc -u -w 1 -p 6665 griffin 6666 # Enforce source port 6665. Option -w 1 to timeout after 1 sec
echo bar | nc -u -w 1 -p 6665 griffin 6666
</source>

=== netstat ===
See [[Linux networking]].

=== nmap ===
'''nmap''' is the network exploration tool and security / port scanner.

<source lang=bash>
nmap localhost # Scan and print all open ports on 'localhost'
nmap -sP hostname # Simply test whether 'hostname' is accessible
nmap -sP 192.168.1.0/24 # Scan all local network with mask 192.168.1.0/24 (sudo: show mac address)
</source>

;Detect host availability
One can use ''nmap'' in a bash script to test whether a given host is accessible, but it turns out that ''ping'' is faster:
<source lang=bash>
# Using nmap
nmap -sP $1 2>/dev/null | grep -q "1 host up" # Too slow, does not always detect hosts

# ... or ping (faster)
if [ "$HOST_MACH" = "cygwin" ]; then
ping $1 1 1 2>/dev/null | grep -q "1 packets received"
else
ping -c 1 $1 2>/dev/null | grep -q "1 received"
fi
</source>

;Detect all Raspberry pi on the network
<source lang=bash>
sudo nmap -sP 192.168.1.0/24 | awk '/^Nmap/{ip=$NF}/B8:27:EB/{print ip}'
sudo nmap -sP 192.168.1.0/24 | grep -B2 Raspberry
</source>

;Various scan
<source lang=bash>
sudo nmap -A -T4 myserver.org # Quick port scan on myserver.org
sudo nmap -A -T4 -p22 myserver.org # Get SSH banner from server
</source>

=== nmcli ===
See [[Linux networking]].

=== nm-tool ===
'''nm-tool''' is an utility to report NetworkManager state and devices.

<source lang=bash>
nm-tool # Print IP addresses, DNS servers, etc.
</source>

=== /etc/init.d/nscd ===
This is not really a command, but the init service ''Name Service Cache Daemon''.
<source lang="bash">
<source lang="bash">
sudo /etc/init.d/nscd restart # To restart daemon and flush DNS cache
netstat -atpn #All, tcp, socket program PID, numeric
netstat -rn #Kernel route table, numberic
</source>
</source>
When listing sockets (default output), you'll get an output like:
{|
|-
|
% netstat -at
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 *:time *:* LISTEN
tcp 0 0 localhost:mysql *:* LISTEN
tcp 0 0 andLinux.local:43449 windows-host:x11 ESTABLISHED
|
% netstat -atn
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:37 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN
tcp 0 0 192.168.11.150:43449 192.168.11.1:6000 ESTABLISHED
|}
:;Local Address
::'''*''' or '''0.0.0.0''' means that the process accepts connection from any interface.
::'''127.0.0.1''' means it only accepts connection on localhost loopback (and so only connection that originates from local PC as well).
::Any other IP address means that the process listen on the given port at the given IP address


== recode ==
=== nslookup ===
'''nslookup''' is a program to query the DNS database. It is available on Linux and Windows, but on Linux is is deprecated by ''dig''.

<source lang=bash>
nslookup 8.8.8.8 # Reverse DNS lookup
</source>

=== rpcinfo ===

'''rpcinfo''' reports RPC information

<source lang="bash">
rpcinfo -p # Probe the portmapper on host, and print a list of all registered RPC programs
</source>

=== [http://www.dest-unreach.org/socat/doc/socat.html socat] ===
Command-line utility that establishes two bidirectional byte streams and transfers data between them ([http://wiki.yobi.be/wiki/Bypass_Proxy#Client_side:_using_socat]). ''socat'' is the more powerful version of ''netcat''. Check the [http://www.dest-unreach.org/socat/ homepage]. And also this page on [http://wiki.yobi.be/wiki/Bypass_Proxy#Client_side:_using_socat Yobi] for examples on how to use '''socat''' to bypass a proxy.

<source lang=bash>
# All these commands listen on localhost:53805, and forward to server:53806
socat TCP4-LISTEN:53805 TCP:server:53806 # Listen, and close as soon socket closes
socat TCP4-LISTEN:53805,reuseaddr,fork TCP:server:53806 # Idem, but fork so that listener continues after socket closes

socat -v ... # Send txfed data to stderr, with some formatting (text format)
socat -x ... # ... idem in hex format (can be combined with -v)
socat -ly # Writes messages to syslog instead of stderr; severity as defined with option -d
</source>

'''socat''' can easily tunnel connections:
<source lang="bash">
socat -ly 'TCP4-LISTEN:143,reuseaddr,fork' PROXY:ton.imap.server:143|TCP:134.27.168.36:8080
</source>

'''socat''' can also be used as ''SSH'' <code>ProxyCommand</code>:
<source lang="bash">
ProxyCommand socat -ly - 'PROXY:%h:%p|TCP:proxy.server:8080'
# Using socat v2.0.0 beta
ProxyCommand socat -ly - 'PROXY:%h:%p,proxyauth=user:pass|SSL,verify=0|PROXY:my.server:443,proxyauth=user:pass|TCP:big.brother.proxy:8080'
</source>

'''socat''' can be easily used as a replacement of ''telnet'':
<source lang="bash">
socat tcp:<host>:<port> - # <port> can be a port number or service name (telnet,imap...)
</source>

To establish '''UDP''' transfer, use:
<source lang="bash">
socat UDP-RECV:[port] STDOUT # On destination host
socat STDIN UDP-DATAGRAM:[host]:[port] # On source host
</source>

<pre>
QUESTION TO MR DOEGOX:
- WHY -s in SED?
- Where does it get info on % for socat?

$ socat echo -
$ socat echo STDIO
$ socat echo STDOUT%STDIN
$ socat echo STDIO%STDIO
$ socat echo -%-
$ socat echo FD:1%FD:0
$ socat echo 1%0

/usr/local/bin/socat TCP:127.0.0.1:1234 -%EXEC:"awk 'BEGIN{print \"BANNER\";fflush()}/BANNER/{next}//{print;fflush()}'"

Testing socat:
- /usr/local/bin/socat - TCP-LISTEN:1234
- /usr/local/bin/socat TCP:127.0.0.1:1234 EXEC:"/bin/sed -s 's/foo/bar/g'"%EXEC:"/bin/sed -u 's/toto/tata/g'"

First open a port for listening and connect it to STDIO
In another shell, connect and do the black magic...


reference:
- http://eros.ph0k.eu/~nizox/twin/V2/modules/socat-1.6.0.0/EXAMPLES
- http://www.rschulz.eu/2008/09/ssh-proxycommand-without-netcat.html
$ exec 3<>/dev/tcp/zeus.yobi.be/22;(cat <&3 & );cat >&3
SSH-2.0-OpenSSH_5.1p1 Debian-3
- http://thesmithfam.org/blog/2006/05/23/bash-socket-programming-with-devtcp-2/
- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464
- http://www.mtu.net/~engstrom/ssh-proxy.php



ATTEMPT 3:
Window 1: socat -v -%'EXEC:/home/beq06659/.ssh/mys2.sh' TCP-LISTEN:1234
Window 2: /usr/local/bin/socat -ly -v -%"EXEC:sed -unf /home/beq06659/.ssh/sedy" "PROXY:ftp.noekeon.org:22|TCP:localhost:1234"

File mys2.sh:
#! /bin/bash
echo -e "HTTP/1.1 200 Connection established\r\n\r\n"
cat
File sedy:
1 s/^/SSH-2.0-OpenSSH_5.1\n/p; /SSH-2.0-OpenSSH_5.1/d; /SSH-2.0-OpenSSH_5.1/! p

LESSONS LEARNED:
- READ THE CHANGES DOC, especially for BETA SW
- SOCKET = bidir --> a filter = 4 descriptors
- PIPE = unidir --> a filter = 2 descriptor (stdin/stdout)
</pre>

=== ss ===
See [[Linux networking]].

=== wavemon ===
'''wavemon''' is a top-like monitoring tool for wifi connection.

To install:
<source lang="bash">
sudo apt install wavemon
</source>

To use:
<source lang="bash">
wavemon
</source>

=== wget ===
<source lang=bash>
wget URL
wget URL --content-dispostion # If download gets wrong name, use this option first
wget URL -O filename # ... otherwise use this to rename download to 'filename'
</source>

=== whois ===
Find CIDR for www.facebook.com [http://www.cyberciti.biz/tips/linux-iptables-examples.html]:
<source lang=bash>
host -t a www.facebook.com
# www.facebook.com has address 69.171.228.40

whois 69.171.228.40 | grep CIDR
# CIDR: 69.171.224.0/19
</source>

== Internet ==

=== httrack ===
'''httrack''' is a offline browser; it copies websites to a local directory.

<source lang=bash>
# Example: checking out progit boot
# -d stay on same principal domain
# -w mirror web sites
# -O /var/www path for mirror/logfiles+cache
# -I0 don't make an index
# -%v display on screen filesnames downloaded
# -P 127.0.0.1:8118 Proxy settings
# '+progit.org/figures/*' +filter
# '-progit.org/book/*/*' -filter
httrack http://progit.org/book/ -d -w -O /var/www -I0 -%v '+progit.org/figures/*' '-progit.org/book/*/*' [-P 127.0.0.1:8118]
sed -ri \"s_'http://progit.org/book'_'index.html'_\" /var/www/progit.org/book/*.html

# Example 2
# -W mirror web sites, semi-automatic (ask question)
httrack http://www.cplusplus.com/ -W -O /var/www -%v '-www.cplusplus.com/forum/*' '-www.cplusplus.com/src/*' '-www.cplusplus.com/member/*' [-P 127.0.0.1:8118]

# Example 3 - Use different path for cache/log files
# -O /var/www,/tmp/httrack path for mirror,path cache and logfiles
httrack http://www.argospress.com/Resources/systems-engineering/index.html -d -O /var/www,/tmp/httrack -I0 -%v [-P 127.0.0.1:8118]
sed -ri '/wms.assoc-amazon.com/{N; d}' /var/www/www.argospress.com/Resources/systems-engineering/*.htm

# Example 4 - Limit depth
# -r2 set mirror depth to 2 (i.e. given address + 1st links)
httrack http://powerman.name/doc/asciidoc -r2 -w -I0 [-P 127.0.0.1:8118]
</source>

== Serial ==
=== stty ===
'''stty''' changes and prints terminal line settings. It usually comes pre-installed.

<source lang=bash>
stty -F /dev/ttyUSB0 -a # Print all settings in human-readable form
stty -F /dev/ttyUSB0 --save # Print all settings in stty-readable form
</source>

=== setserial ===
'''setserial''' get/set Linux serial port information.

See for instance [http://accesio.com/go.cgi?p=../software/linuxserial.html] for usage.

== Package Management ==

=== dpkg ===
Package manager for Debian / Ubuntu. See [[Package Management#Ubuntu / Debian]]

=== rpm ===
See [[Package Management#RPM's]]

== System ==

=== dmidecode ===
'''dmidecode''' is a tool to decode DMI (SMBIOS) tables:
<source lang="bash">
sudo dmidecode
</source>
<source lang="text">
...
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Dell Inc.
Version: A19
Release Date: 12/21/2009
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 1728 kB
Characteristics:
ISA is supported
PCI is supported
PC Card (PCMCIA) is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
...
</source>

=== inxi ===
'''inxi''' is yet another way to get information on hardware:

<source lang=bash>
inxi -Gx # as current desktop user, don't use sudo!
# Graphics: Card: Intel 3rd Gen Core processor Graphics Controller bus-ID: 00:02.0
# X.Org: 1.15.1 driver: intel Resolution: 1600x900@60.0hz
# GLX Renderer: Mesa DRI Intel Ivybridge Mobile GLX Version: 3.0 Mesa 10.1.0 Direct Rendering: Yes
</source>

=== loadkeys ===
'''loadkeys''' loads keyboard translation tables.

Use ''loadkeys'' to change the keyboard layout in console mode (i.e. {{kb|Ctrl-Alt-F1}} console).

<source lang=bash>
sudo loadkeys be # Load Belgian layout
</source>

=== lscpu ===
<source lang="bash">
lscpu # Display information about the cpu architecture
</source>

=== lsmem ===
<source lang="bash">
lsmem # List all ranges of available memory
</source>

=== lsmod / modinfo / modprobe ===
<source lang="bash">
lsmod # Show the status of modules in the Linux Kernel
modinfo iwlagn # Show information about a Linux Kernel module
modprobe -r iwlagn # Remove module from the Linux Kernel
modprobe iwlagn 11n_disable=1 # Add module to the Linux Kernel, with specific parameters
</source>

=== lspci ===
<source lang="bash">
lspci # List all PCI devices
lspci -v # List all PCI devices (verbose)
</source>

=== newgrp ===
'''newgrp''' log in to a new group

<source lang=bash>
newgrp - <group> # Start a new session with <group> as primary group. - means to reinitialize environment.
umask 002 # Make sure all files / directories we create have group access
</source>

To run ''newgrp'' in a script, use the following construction [http://stackoverflow.com/questions/3735858/change-default-group-in-script]:
<source lang=bash>
newgrp groupb << END
touch "myscript_output.txt"
END
</source>

Or better yet use the [[#sg|sg]] command instead.

=== pidinfo ===
Find the process ID of a running program
<source lang="bash">
pidof apache2 # Find all pid of apache2
</source>

=== pgrep ===
Prints PID of the process matching given pattern. Never match pgrep itself!
<source lang="bash">
ps -fp $(pgrep autossh) # Much better that "ps -aef | grep autossh", which match grep process itself
</source>

=== pkill ===
Send a signal (default TERM) to process matching given pattern. Never match pkill itself!
<source lang="bash">
pkill -0 autossh # Test if process "autossh" is running
</source>

=== ps ===
<source lang="bash">
ps -ef # All processes - full format

ps afl # Proc w ttys, all users - long + forest
ps flx # Proc w/o ttys - long + forest
ps aflx # All processes, all users - long + forest

ps au # Proc w ttys, all users - user output
ps ux # Proc w/o ttys - user output
ps aux # All processes, all users - user output

ps axs # All process, with signals
</source>

=== [http://www.ivarch.com/programs/pv.shtml pv] ===
'''pipe viewer''' &mdash; terminal-based tool for monitoring the progress of data through a pipeline

<source lang="bash">
pv -cN source < linux-2.2.20.tar.bz2 | bzcat | pv -CN bzcat | gzip -9 | pv -cN gzip > linux-2.2.20.tar.gz
</source>
<source lang="text">
source: 4.48MB 0:02:01 [9.84kB/s] [========> ] 29% ETA 0:04:44
bzcat: 26MB 0:02:01 [ 192kB/s] [ <=> ]
gzip: 5.58MB 0:02:01 [ 64kB/s] [ <=> ]
</source>

=== savelog ===
'''savelog''' saves, rotates and compresses log files. This is similar to logrotate cron job.

<source lang="bash">
savelog -n ~/log/git-fetch.log
</source>

=== sg ===
'''sg''' executes command as different group ID.

<source lang=bash>
umask 002 # Ensure that all files written will be group accessible
sg - <group> -c "mkdir foo" # Run commands as group <group>
</source>

The following construction restart the current script under a different group (from [http://stackoverflow.com/questions/3735858/change-default-group-in-script]):
<source lang=bash>
if ! [ "${SBREADY:=false}" = true -o $(id -ng) = ${NEEDGRP:=wheel} ] ; then
export SBREADY=true
exec sg $NEEDGRP "$0" "$@"
fi
</source>

=== sudo / gksudo ===
See also Ubuntu's page at [linux reference script security sudo ubuntu].

<source lang="bash">
ls | sudo tee /root/somefile # Instead of redirection sudo ls > /root/somefile
ls | sudo tee -a /root/somefile # ... same but append instead
sudo sh -c "ls > /root/somefile" # Yet another way to do redirection with sudo
sudo !! # Repeat last command with root privilege
sudo -i # Get a root *login* shell
sudo su - # ... another solution
sudo -s # Root shell but keeping current environment
sudo su # ... another solution
sudo -k # Reset sudo password timeout
</source>

'''gksudo''' is the graphical equivalent of '''sudo''':
<source lang="bash">
gksudo gedit /etc/fstab
</source>

=== showkey ===
'''showkey''' examine the codes sent by the keyboard (linux kernel only - for X keyboard events, see '''[[X|xev]]'''!).
<source lang="bash">
showkey -k # Dump key codes (default)
showkey -s # Dump scan codes
</source>

=== timeout / timelimit ===
'''timeout''' / '''timelimit''' are small binary to run a command with a given time limit. More exactly they send a given signal to the child process after some specified time. See also [[Perl#Kill on ALARM signal|Perl - kill on ALARM signal]]).

<source lang="bash">
timeout -15 30 mplayer dvd:// # Show 30s of DVD movie, then kill mplayer with a TERM signal
</source>

=== Memory related ===
;free
:Display the amount of free and used memory ('''free -m''' to get info in MB)
<source lang="bash">
free
free -m # Show in megabyte
</source>
;<tt>/proc/meminfo</tt>
:All memory usage information
<source lang="bash">
cat /proc/meminfo
</source>
;vmstat
:Summary of memory usage
<source lang="bash">
vmstat
</source>

=== Miscellaneous ===
;[http://htop.sourceforge.net/index.php htop]
:an improved ''top'' command
;lsb_release
:'''lsb_release -a''' prints version information for the Linux release you're running.
:Another option (also working on Debian), is to use <code>cat /etc/*-release</code>.
;mkfifo
:make FIFOs (named pipes)
;pv, pipeview, pipebench
:monitor the progress of data through a pipe
;strace
:trace system calls and signals
;tee
:read from standard input and write to standard output and files
;uname
:'''uname -a''' prints all system information, include machine name, kernel name & version.
;watch
:Execute a program periodically, showing output full screen
<source lang="bash">
watch -n 0.5 myprog arg1 arg2 # Wait 0.5s between run
watch -cn 0.5 myprog arg1 arg2 # ... same with color
watch -cn 0.5 myprog \"arg w/ space\" # BEWARE! arg must be "double" quoted
watch -cn 0.5 myprog ${*@Q} # Quoted args, in case args are user defined in a script
</source>

== User / Group Administration ==

=== groupadd / addgroup ===
Use '''groupadd''', '''addgroup''' to create a new group. '''groupadd''' is the low-level utility. '''addgroup''' is the Debian more friendly version.

<source lang="bash">
sudo groupadd groupname # Create a new group groupname
sudo groupadd -g 123 groupname # ... and specify gid of the new group
sudo addgroup groupname
sudo addgroup --gid 123 groupname
</source>

=== useradd / adduser ===
Use '''useradd''' to create new user. '''useradd''' is the low-level utility. '''adduser''' is the Debian more friendly version.

<source lang="bash">
sudo useradd username # Create a new user username
sudo useradd -s /bin/bash username # ... specify the deafult shell in use for the user
sudo useradd -s /bin/bash -m username # ... and create user home directory (from /etc/skel)
sudo useradd -u 1234 username # Create a new user username with uid 1234
sudo useradd -g GRP1 -G GRP2,GRP3 username # Specify primary and supplementary groups
sudo adduser username # Create a new user username
sudo adduser --system username # Create a new system user (in the system uid range)
sudo adduser --shell /bin/bash username # Use /bin/bash as default login shell
</source>

=== userdel / groupdel / deluser ===
Use '''userdel''', '''groupdel''', '''deluser''' to remove an existing user /group. '''deluser''' is the Debian more friendly version

<source lang="bash">
sudo userdel username # Remove existing user username
sudo groupdel groupname # Remove existing group groupname
sudo deluser username # Remove existing user username
sudo deluser --group groupname # Remove existing group groupname
</source>

=== usermod ===
Use '''usermod''' to modify an existing user

<source lang="bash">
sudo usermod -g GRP1 username # Modify the primary group of user username
sudo usermod -g GRP1 -G GRP2,GRP3 username # ... and also the supplementary groups
</source>

=== gpasswd ===
Use '''gpasswd''' to administer /etc/group and /etc/gshadow

<source lang="bash">
sudo gpasswd -a $USER fuse # Add self to group 'fuse'
</source>

To make the new group effective:
<source lang=bash>
loginctl terminate-user $USERNAME # This will FORCE LOGOUT !!!
su - $USERNAME # New shell with new groups
</source>

== X ==

=== [http://willem.engen.nl/projects/disper/ disper] ===
TO install:
<source lang="bash">
sudo add-apt-repository ppa:wvengen/ppa
sudo apt-get update
sudo apt-get install
</source>

Some examples of use:
<source lang="bash">
disper -s # Only enable primary screen (laptop screen)
disper -S # Only enable secondary screen (external monitor)
disper -d auto -e # Auto-detect + enable extended mode
disper -d auto -c # clone mode
</source>

=== xbindkeys / xbindkeys-config ===
* '''xbindkeys''' is a program that allows you to launch shell commands with your keyboard or your mouse under the X Window System. It links commands to keys or mouse buttons, using a configuration file. It's independent of the window manager and can capture all keyboard keys (ex: Power, Wake...).
* '''xbindkeys-config''' is an easy to use gtk program for configuring Xbindkeys (see [http://perens.com/works/software/gyration.html].

=== [http://freshmeat.net/projects/xclip/ xclip] ===
:xclip is a command line interface to the X11 clipboard. It can also be used for copying files, as an alternative to sftp/scp, thus avoiding password prompts when X11 forwarding has already been setup. [http://gnuisance.net/blog/t/2007/xclip.html Check this guide].

=== xdpyinfo ===
Display information utility for X, like the dimension of the display (pixels vs millimeters) and the resolution (DPI):

<source lang="bash">
xdpyinfo | grep dimensions # Return dimension of the desktop (size in pixels and millimeters)
xdpyinfo | grep resolution # Return current screen resolution in DPI (dots per inch)
</source>

=== XSel / XSelection ===
'''[http://www.vergenet.net/~conrad/software/xsel/ XSel]''' is a command-line program for getting and setting the contents of the X selection and context-menu clipboard. Normally the former is only accessible by manually highlighting information and pasting it with the middle mouse button, and the latter is only accessible by using context-menu cut/copy/paste command.

'''xselection''' is an equivalent package on OpenSUSE.

On [[Cygwin]], one has to use <tt>getclip</tt>, <tt>putclip</tt> or device <tt>/dev/clipboard</tt>.

<source lang="bash">
xsel -p # To get content of PRIMARY buffer (X selection)
xselection PRIMARY # (equivalent command for xselection)
xsel -b # To get content of CLIPBOARD (context-menu clipboard)
xselection CLIPBOARD # (equivalent command for xselection)
echo sometext | xsel -p # To set PRIMARY buffer
echo sometext | xselection PRIMARY - # (equivalent command for xselection)
xselection PRIMARY sometext # (equivalent command for xselection)
echo sometext | xsel -b # To set CLIPBOARD buffer
echo sometext | xselection CLIPBOARD - # (equivalent command for xselection)
xselection CLIPBOARD sometext # (equivalent command for xselection)
</source>

=== xvkbd ===
'''xvkbd''' is a virtual (graphical) keyboard program for X Window System which provides facility to enter characters onto other clients (softwares) by clicking on a keyboard displayed on the screen. It can also be used to send keyboard events through command-line:
:<source lang="bash" enclose="prevalid">
xvkbd -xsendevent -text text-string
</source>

=== xwininfo ===
'''xwininfo''' is the window information utility for X. Could be used to find the geometry of a window. See [http://www.debian-administration.org/article/Finding_the_dimensions_of_the_X11_display here]
<source lang="bash">
xwininfo # Return information on the clicked windows
xwininfo -root # ... or on the root window
xwininfo -root |awk '/Width/{print $2} # Get Width of the current desktop
</source>

== Text Manipulation ==

=== column ===
Format input into multiple columns

<source lang="bash">
cat | column -l
123 456
98765 98765
^D
# 123 456
# 98765 98765
</source>

=== csvquote ===
'''[https://github.com/dbro/csvquote csvquote]''' is a nice wrapper tool to pre-/post-process CSV files to ease processing by tools like AWK (to handle comma within quotes)

<source lang="bash">
csvquote foobar.csv | cut -d ',' -f 5 | sort | uniq -c | csvquote -u
cat foobar.csv | csvquote | cut -d ',' -f 7,4,2 | csvquote -u
csvquote -t foobar.tsv | wc -l
csvquote -q "'" foobar.csv | sort -t, -k3 | csvquote -u
csvquote foobar.csv | awk -F, '{sum+=$3} END {print sum}'
</source>

=== cut ===
TBC

=== paste ===
TBC

=== fold ===
<source lang="bash">
fold -w 40 FILE # Wrap FILE to 40-char width
</source>

=== expand ===
<source lang="bash">
expand -t 4 FILE # Convert tabs to spaces (4)
expand -t 4,10 FILE # Set explicit tab positions
</source>

=== fmt ===
TBC

=== less ===
<source lang=bash>
command | less -S # To wrap (aka fold) long-lines (or use '-','S' within less)
</source>

=== nl ===
'''nl''' numbers lines of files
<source lang=bash>
nl -ba FILE # Number *all* lines of FILE
</source>

=== rl, shuf ===
'''rl''' is a command-line tool that reads lines from an input file or stdin, randomizes the lines and outputs a specified number of lines (requires package '''randomize-lines''').

'''shuf ''' generates random permutations.


They are alternative to <code>sort -R</code> (note that the latter does not work with duplicate lines).

=== sed ===
Moved to page dedicated to [[Sed]].

=== sort ===
'''sort''' is the well-know text sort utility.

<source lang=bash>
sort -R shuffle.txt # shuffle the lines (use random hash) - fail with duplicate lines
nl -ba file | sort -R | sed 's/^.*[0-9]\t//' # ... idem, but workaround for dup lines
</source>

<code>sort</code> by default takes into account the current ''locale''. To get the traditional sort order that uses native byte values:
<source lang="bash">
LC_ALL=C sort myfile.txt # traditional sort, indep of current locale
</source>

Use '''<code>-o file</code>''' to sort in-place, or use the '''<code>sponge</code>''' utility:
<source lang="bash">
sort file -o file # sort in-place
sort file | sponge file # soak sorted file and write it back
</source>

Use <code>LC_ALL=C</code> for '''huge speed improvement'''. Also increasing the internal buffer helps:
<source lang="bash">
LC_ALL=C sort -S 2G file
</source>

=== tr ===
'''tr''' translates or deletes characters.

Some basic uses:
<source lang=bash>
echo lowercase | tr a-z A-Z # turns lowercase into uppercase
echo lowercase | tr [:lower:] [:upper:] # idem
echo encrypt me please | tr a-z n-za-m # a very simple rot13 implementation
</source>

A powerful rot-n implementation [http://stackoverflow.com/questions/6441260/how-to-shift-each-letter-of-the-string-by-a-given-number-of-letters]:
<source lang=bash>
dual=abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
phrase='hello there'
n=13
newphrase=$(echo $phrase | tr ${dual:0:26} ${dual:$n:26})
echo ${newphrase}
</source>

tr can also be used to split words into newlines:
<source lang=bash>
tr ' ' '\n' file
</source>

=== figlet, toilet... (fun) ===
Some funny banner-like tool:
* <code>figlet</code> [http://www.figlet.org/]
* <code>toilet</code> [http://caca.zoy.org/wiki/toilet]
* <code>cowsay</code>
* <code>ponysay</code>
* <code>lolcat</code>

== Image manipulation ==
=== netpbm ===
{{deb|Netpbm}} is a package of graphics programs and programming libraries. There are over 220 separate programs in the package, most of which have <tt>pbm</tt>, <tt>pgm</tt>, <tt>ppm</tt>, or <tt>pnm</tt> in their names. For example, <code>pnmscale</code> and <code>giftopnm</code>.

For example, you might use <code>pnmscale</code> to shrink an image by 10%. Or use <code>pnmcomp</code> to overlay one image on top of another. Or use <code>pbmtext</code> to create an image of text. Or reduce the number of colors in an image with <code>pnmquant</code>.

<source lang=bash>
anytopnm myfile.gif | pnmtopng > myfile.png # Convert a GIF file to PNG
anytopnm oll28.gif | pnmflip -r90 | ppmtogif > oll28-90.gif # Rotate and output GIF
</source>

JPEG file manipulation can be done with <code>djpeg</code> and <code>cjpeg</code> from package {{deb|libjpeg-turbo-progs}}:

<source lang="bash">
djpeg foo.jpg | pnmflip -r90 | cjpeg) # Lossy rotate of jpeg
</source>

== Development ==

=== addr2line ===
'''addr2Line''' converts addresses into file names and line numbers.

=== ar ===
'''ar''' creates, modifies, and extracts from archives. It can be used to generate static libraries (<tt>.a</tt> files) for use with ''gcc''.

=== diff ===
Use '''diff''' to compare 2 files/directories together and/or to generate patch files. Use '''colordiff''' (package <tt>colordiff</tt>) to have colored output.!
<source lang="bash">
diff old new # View difference (default ed script format)
diff -u old new >new.patch # Generate a patch file (universal format)
diff -y old new # View difference side-by-side (as with sdiff)

#Color with DIFF:
colordiff -u old new # colordiff can also colorize a diff output package colordiff
diff old new | colordiff # ... idem

#Color with WDIFF:
wdiff -n old new | colordiff # But need option '-n'

#Piping to LESS with COLORS
diff -u old new | colordiff | less -R # To keep color with pager

#DIFF / WDIFF combo
diff --strip-trailing-cr -bu old new | wdiff -d | colordiff | less -R # Handy when wdiff miss some options...
</source>

Patch can be applied with the command '''patch''' (see below). Be careful regarding file names / directory structures when generating patch files! Sometimes you'll need to edit the generated patch files to fix the directory structure.

=== ldd ===
'''ldd''' print shared object dependencies.

See also <code>readelf</code> and <code>lddtree</code> (package {{deb|pax-utils}}).

<source lang="bash">
ldd /bin/ls
# linux-vdso.so.1 (0x00007ffcc3563000)
# libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f87e5459000)
# libcap.so.2 => /lib64/libcap.so.2 (0x00007f87e5254000)
# libc.so.6 => /lib64/libc.so.6 (0x00007f87e4e92000)
# libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f87e4c22000)
# libdl.so.2 => /lib64/libdl.so.2 (0x00007f87e4a1e000)
# /lib64/ld-linux-x86-64.so.2 (0x00005574bf12e000)
# libattr.so.1 => /lib64/libattr.so.1 (0x00007f87e4817000)
# libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f87e45fa000)
</source>

=== lddtree ===

lddtree


=== readelf ===
'''readelf''' displays information on ELF files.

<source lang="bash">
# View libraries linked in .elf (but only directly linked, these may yet load other libs)
readelf -d obs-ios-camera-source.so | grep NEEDED
# 0x0000000000000001 (NEEDED) Shared library: [libobs.so.0]
# 0x0000000000000001 (NEEDED) Shared library: [libavcodec.so.58]
# ...
</source>

=== patch ===
Use '''patch''' to apply a patch previously generated with the command '''diff'''. '''patch''' use is pretty much straightforward, one mainly has to pay attention to directory structure / blanks.

Complete example using '''diff''':
<source lang="bash">
$ cp project project-patched
$ cd project-patched
$ vi somefile # We start modifying the copy
$ cd ..
$ diff -u project project-patched >project.patch # We generate the patch file (universal format)
$ cd project # We cd into project directory to patch
$ patch -lNp1<../project.patch # We apply the patch in place
</source>
* '''<tt>-l</tt>''' &mdash; ignore white spaces (very handy when copying patch from internet page for instance).
* '''<tt>-N</tt>''' &mdash; ignore patch that seems reversed or already applied (same as <tt>--forward</tt>).
* '''<tt>-p1</tt>''' &mdash; tells '''patch''' to ignore the first (i.e. one level of) directory in the file name given in the patch header.
The value to use for option <tt>p</tt> depends actually on the patch header:
<ul>
<li> First example - same root directory for both to and from file</li>
<source lang="bash">
--- outguess-0.2/jpg.c 2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2/jpg.c 2009-08-25 16:06:05.242378300 +0200
</source>

<source lang="bash">
$ patch -lNp0 <project.patch # First example
</source>

<li> second example - different root directory between to and from file</li>
<source lang="bash">
--- outguess-0.2/jpg.c 2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2-patched/jpg.c 2009-08-25 16:06:05.242378300 +0200
</source>

<source lang="bash">
$ cd project_dir; patch -lNp1 <../project.patch # Second example
</source>
</ul>

=== vbindiff ===
'''vbindiff''' &mdash; hexadecimal file display and comparison

<source lang=bash>
vbindiff file1.bin file2.bin # Compare and view 2 binary files
</source>

== Scripting ==
Some commands very useful when writing scripts

=== command ===
From the manpage, <code>command</code> execute a simple command...

<source lang=bash>
GDB=$(command -v gdb) # Get path to gdb
</source>

<code>command -v</code> behaves much like <code>which</code>, except that on Bash:
* <code>command</code> is a shell built-in.
* it treats aliases differently.

== Miscellaneous ==

=== base64 ===
'''base64''' from coreutils

<source lang="bash">
echo QWxhZGRpbjpvcGVuIHNlc2FtZQ== | base64 --decode
echo `echo QWxhZGRpbjpvcGVuIHNlc2FtZQ== | base64 --decode`
</source>


=== cksum / jacksum / md5sum / sha1sum / sum ===
<source lang="bash">
cksum file # Generate a CRC similar to CRC32, but in decimal, and with length appended.
cksum -o3 file # Generate a CRC32 checksum (BSD only)
sum file # Generate a checksum
md5sum file # Hash file using MD5
sha1sum file # Hash file using SHA1
jacksum -a crc32 file # Hash using CRC32 (dec)
jacksum -a crc32 -E hexup file # Hash using CRC32 (hex)
</source>

=== date ===

<source lang=bash>
date +'%F %T' # YYYY-mm-dd HH:MM:SS
date +'%Y%m%d%H%M%S" # YYYmmddHHMMSS
date +'%s' # Get current epoch time (seconds since 1970-01-01 00:00:00 UTC)
date --date='@1498485311' # Convert epoch to human readable time
</source>

=== direnv ===
[https://direnv.net/ direnv] is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory.

* Alternatives: https://github.com/gyf304/dotenv

<source lang="bash">
# Create a new folder for demo purposes.
$ mkdir ~/my-project
$ cd ~/my-project

# Show that the FOO environment variable is not loaded.
$ echo ${FOO-nope}
nope

# Create a new .envrc. This file is bash code that is going to be loaded by
# direnv.
$ echo export FOO=foo > .envrc
.envrc is not allowed

# The security mechanism didn't allow to load the .envrc. Since we trust it,
# let's allow its execution.
$ direnv allow .
direnv: reloading
direnv: loading .envrc
direnv export: +FOO

# Show that the FOO environment variable is loaded.
$ echo ${FOO-nope}
foo

# Exit the project
$ cd ..
direnv: unloading

# And now FOO is unset again
$ echo ${FOO-nope}
nope
</source>

=== dog ===
'''dog''' is better than '''cat'''. It writes the contents of each give file, URL, or the standard input to standard output.

=== dtach ===
'''dtach''' simple program that emulates the detach feature of screen. A session in '''dtach''' is a single instance in which a program is running under the control of '''dtach'''. The program is disassociated from the original terminal, and is thus protected from your original terminal being disconnected for some reason.

=== echo ===
<source lang="bash">
echo -e "Some text\n...on 2 lines..." # Enable interpretation of backslash escapes (must be quoted!)
</source>

=== [http://expect.nist.gov/ expect] ===
Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.
* See [[SMTP#Expect script for testing SMTP]]

=== hyperfine ===
'''[https://github.com/sharkdp/hyperfine hyperfine]''' is a command-line benchmarking tool (benchmark tool).

<source lang="bash">
hyperfine --warmup 3 'fd -HI "[0-9]\.jpg"' 'find -name "*[0-9].jpg"'
</source>

Hyperfine may produce results in CSV, JSON, Markdown.

=== iconv ===
locale encoding conversions

=== konwert ===
fancy encoding conversions

=== [[Midnight Commander|mc (Midnight Commander)]] ===
'''mc''' or '''Midnight Commander''' is a powerful file manager working in a shell terminal.<br/>
See [[Midnight Commander|dedicated page]] on this wiki.

=== mimencode ===
binary file conversion for the mail.

=== munpack ===
Use munpack to extract MIME attachment from email [https://superuser.com/questions/406125/utility-for-extracting-mime-attachments]:

<source lang="bash">
cat Message
# ...
#
# --PGP_Universal_31BF743E_DCBDB54C_D4397473_15481C77
# Content-Type: application/octet-stream;
# name="tuleap_ex.txt.gpg"
# Content-Transfer-Encoding: BASE64
# Content-Disposition: attachment;
# filename="tuleap_ex.txt.gpg"
#
# hQIOA1uCgmTNqsbUEAgAqbp/CXJfNQmZu8RHtZaP6Fm5gi+sjyothO19faktgOFrVTsame5ZW8yi
# ...
# ...
# NnRRvRI+TcohkYsisQy4oiUA
# --PGP_Universal_31BF743E_DCBDB54C_D4397473_15481C77--
#
sudo apt install mpack
munpack -f Message
</source>

=== mrxvt ===
A multi-tab version of rxvt (terminal-emulator).

=== parallel ===
'''parallel''' runs many job in parallel.

<source lang="bash">
parallel -i chronic sh -c "date; echo {}"
</source>

Note that this is <code>parallel</code> from {{deb|moreutils}}. There is also a separate package {{deb|parallel}} with different options.

=== recode ===


<source lang="bash">
<source lang="bash">
Line 99: Line 1,717:
|-
|-
| '''u2''' || UCS-2 (Universal Character Set, 2 bytes)
| '''u2''' || UCS-2 (Universal Character Set, 2 bytes)
| '''/64''' || Base64 encoded text
| '''/64''' || Base64 encoded text (see also [[#base64|base64]]) from coreutils.
|-
|-
| '''u8''' || UTF-8 (Universal Transformation Format, 8 bits)
| '''u8''' || UTF-8 (Universal Transformation Format, 8 bits)
Line 105: Line 1,723:
|}
|}


== rpm ==
=== reformime ===
MIME E-mail reformatting tool
See [[Package Management#RPM's]]


== sed ==
=== screen ===
'''Screen''' is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.
=== References ===
* [http://sed.sourceforge.net/ The SED Homepage on SourceForge]
* [http://sed.sourceforge.net/sedfaq.html The SED FAQ]
* [http://www.opengroup.org/onlinepubs/007908799/xcu/sed.html The SED man page]
* [http://www.gnu.org/software/sed/manual/sed.html Sed, a stream editor]
* [http://sed.sourceforge.net/sed1line.txt The sed one-liners]


See also ''[[Linux Commands#tmux|tmux]]'' for a better alternatives that supports splitting in panes, mousewheel support...
=== Usage ===

Some basic usage:
Launch with
<source lang=bash>
screen # Start a new screen session
screen -r # Resumes a detacked screen session
screen -d -r # Reattach a session and if necessary detach it first
screen -d -R # Reattach a session and if necessary detach or even create it first
screen -D -r # Reattach a session. If necessary detach and logout remotely first
screen -D -R # Attach here and now, ie:
# - If no session is running, create it
# - Otherwise reattach the session. If necessary detach/logout remotely first
</source>

For convenience one may add the following alias to <tt>.bashrc</tt>:
<source lang=bash>
alias screen=screen -D -R
</source>

Useful shortcuts:
{| class="wikitable"
|-
!key!!description
|-
|{{kb|C-a}} {{kbkey|c}}
|Open a new instance
|-
|{{kb|C-a}} {{kbname|Space}}
|Switch to next instance
|-
|{{kb|C-a}} {{kbkey|d}}
|Detach from screen. Reattach later with <code>screen -r</code>
|-
|{{kb|C-a}} {{kbkey|D D}}
|''Power detach'', i.e. detach and logout
|}

'''[https://help.ubuntu.com/10.04/serverguide/byobu.html byobu]''' is a wrapper around ''screen'', which makes it prettier and more powerful (see [http://kmandla.wordpress.com/2010/01/03/byobu-a-shortcut-to-a-prettier-screen/ this] also).

Use '''[https://github.com/nelhage/reptyr reptyr]''' to capture an already running process and attach it to a new terminal. Very handy if you start a process, and you regret not having started ''screen'' first...

=== sponge ===
'''sponge''' (package ''moreutils'') soaks up standard input and write to a file. It can be used to easily edit file in-place:

<source lang=bash>
sed -r '...' FILE | grep ... | sponge FILE # Pipeline from and to same file!
</source>

=== tabs ===
Set tabulation size of the terminal.
<source lang=bash>
tabs 4 # Set default tab size to 4
tabs 8 # Set default tab size to 8 (the default)
</source>

Note that some applications are not impacted by this setting or have alternate solution:
<source lang=bash>
less -x1,5 # LESS - sets tabs
git config --global core.pager 'less -x1,5' # GIT
diff -t --tabsize=4 # DIFF - alternate solution
</source>

=== tesseract ===
A tool to perform OCR.

Example of script [https://askubuntu.com/questions/280475/how-can-instantaneously-extract-text-from-a-screen-area-using-ocr-tools AskUbuntu]:
<source lang="bash">
<source lang="bash">
#!/bin/bash
sed [OPTION]... {script-only-if-no-other-script} [input-file]...

sed -n # Silent - suppress automatic printing of pattern space
SCR_IMG=".screentemp.png"
sed -r # Use extended regular expression
TEMP_TXT=".screentext.txt"
sed -i "s/foo/bar/" *.txt # In-place file modification

gnome-screenshot -a --file=$SCR_IMG
tesseract $SCR_IMG $TEMP_TXT -l eng
cat $TEMP_TXT* | xsel -b

rm $SCR_IMG $TEMP_TXT*
</source>
</source>


=== timeout ===
Use of address commands <tt>a\text</tt>, <tt>i\text</tt>, <tt>c\text</tt>. The command is terminated by a '''*newline*'''. To insert a newline character, use <tt>\n</tt>:
<code>timeout</code> runs a command with a time limit.

On MinGW, timeout is not available. Instead there is a [https://github.com/dscho/msys/blob/master/msys/packages/bash/2.04/examples/scripts/timeout bash-script equivalent].

=== tmux ===
See page [[Tmux]].

=== uuencode, uudecode ===
Binary file conversion for Unix.

=== wc ===
Counts lines, words and character in a file

=== xargs ===
For more examples, see combinations with the command [[#find|find]]

<source lang="bash">
<source lang="bash">
#By default, xargs cuts at WHITE SPACE and NEWLINE. Use -d '\n' to only cut at NEWLINES
$ cat mytext
for i in $(locate .pdf); do basename $i; done # WRONG
First line
locate .pdf | xargs -rd '\n' -n 1 basemane # Correct
Second line
</source>
$ cat mysedscript
1 {i\inserted text
s/$/ (not anymore)/g}
$ sed -f mysedscript mytext
inserted text
First line (not anymore)
Second line


== Hex Tools ==
# All on one line: use echo -e to generate the newline that terminates the command i\
$ echo -e "1 {i\\inserted text\ns/$/ (not anymore)/g}"| sed -f - mytext
inserted text
First line (not anymore)
Second line


=== hd / hexdump ===
#Same result without command \i:
ASCII, decimal, hexadecimal, octal dump.
$ sed "1 {s/^/inserted text\n/; s/$/ (not anymore)/}" mytext
<source lang="bash">
hexdump -e '"%2x"' <myfile> # Convert myfile into a long hexadecimal string - ! See DOUBLE-QUOTED parameter
hexdump -C <myfile> # Canonical hex + ascii display (16 bytes)
hd <myfile> # (idem)
</source>
</source>
See also [[Linux Commands#od|od]] and [[Linux Commands#xxd|xxd]]. There are also hex editors, like [http://sourceforge.net/projects/hexered/ hexer] and [http://www.physics.ohio-state.edu/~prewett/hexedit/ hexedit] (adapts to width of the terminal!).


=== Regular expressions===
=== od ===
'''od''' - dump files in octal and other formats
The information below is only illustrative. See e.g. [[wikipedia:Regular_expression|Wikipedia page ]] for reference information. The list below is actually for ''extended regular expressions'', which can be obtained in ''sed'' using option '''-r''' (<tt>sed -r</tt>).
<source lang="bash">
{| class="wikitable"
od -xc file # Output file in ascii and hex output (2-byte, ! little-endian)
!Regexp!!Description
od -t x1 -a --width=32 testvar.sh # Output 32-byte at once
|-
</source>
|<tt>.</tt>||Match any character
See also [[Linux Commands#hd / hexdump|hd / hexdump]] and [[Linux Commands#xxd|xxd]].
|-

||<tt>gray|grey</tt>||Match ''gray'' or ''grey''
=== xxd ===
|-
Make a hexdump or do the reverse. This tool is delivered along with '''vim'''. Probably the '''BEST''' out there
||<tt>gr(a|e)y</tt>||Match ''gray'' or ''grey''

|-
<div style="padding-left:2em;"><source lang="bash">
||<tt>gr[ae]y</tt>||Match ''gray'' or ''grey''
xxd -g8 -c32 <file> # Output 32 bytes, grouped in 8-byte columns
|-
xxd -p -c64 <file> # Output 64 bytes per line, plain (postscript) mode
||<tt>file[^0-2]</tt>||Match ''file3'' or ''file4'', but not ''file0'', ''file1'', ''file2''.
echo 202122 | xxd -r -p # Convert hexdump to a binary string
|-
echo -n ' !"' | xxd -p # Convert binary string to hexdump - DON'T FORGET -n
|<tt>colou?r</tt>||''(zero or one)'' - Match ''Color'' or ''Colour''.
echo -n ' !"' | xxd -u -p # Convert binary string to hexdump - DON'T FORGET -n (uppercase)
|-
</source></div>
|<tt>ab*c</tt>||''(zero or more)'' - Match ''ac'', ''abc'', ''abbc'', ....
See also [[Linux Commands#hd / hexdump|hd / hexdump]] and [[Linux Commands#od|od]].
|-

|<tt>ab+c</tt>||''(one or more)'' - Match ''abc'', ''abbc'', ''abbbc'', ....
'''xxd''' can be easily used in Vim to hexedit a file:
|-
<source lang=vim>
|<tt>a{3,5}</tt>||''(at least m and not more than n times)'' - Match ''aaa'', ''aaaa'', ''aaaaa''.
" To convert current buffer to hex
|-
:%xxd
|<tt>^on single line$</tt>||''(start and end of line)'' - Match ''on single line'' on a single line.
" To convert current buffer from hex
|}
:%xxd -r
<small>When using standard (non-extended) regular expression, some special meta-characters (like the parenthesis '''( )''', or braces '''{ }''') must be quoted with '''backslash \'''.</small>
</source>


== Security ==
== [http://www.dest-unreach.org/socat/doc/socat.html socat] ==
=== openssl rand ===
Command-line utility that establishes two bidirectional byte streams and transfers data between them ([http://wiki.yobi.be/wiki/Bypass_Proxy#Client_side:_using_socat]). ''socat'' is the more powerful version of ''netcat''. Check the [http://www.dest-unreach.org/socat/ homepage].
'''openssl rand''' generates random strings:
<source lang="bash">
<source lang="bash">
openssl rand -hex 16 # Generate a 16-byte random string in hexadecimal
socat -ly 'TCP4-LISTEN:143,reuseaddr,fork' PROXY:ton.imap.server:143|TCP:134.27.168.36:8080
openssl rand -base64 16 # Generate a 16-byte random string in base64
ProxyCommand socat - 'PROXY:%h:%p,proxyauth=user:pass|SSL,verify=0|PROXY:my.server:443,proxyauth=user:pass|TCP:big.brother.proxy:8080' # Using v2.0.0 beta
</source>
</source>

''socat'' can be easily used as a replacement of ''telnet'':
== Desktop ==
=== notify-send ===
'''notify-send''' is a program to send desktop notifications.

<source lang=bash>
notify-send "summary" "body message"
</source>

If messages are not visible, use either <code>-t 0</code> to remove time limit, or <code>critical</code> urgency [https://askubuntu.com/questions/187022/how-can-i-send-a-custom-desktop-notification]:
<source lang=bash>
notify-send -u critical "Urgent" "This is urgent!"
notify-send -t 0 "Nag" "This is a nag screen."
</source>

<code>notify-send</code> uses icons from {{file|/usr/share/icons/gnome/32x32}} and {{file|/usr/share/notify-osd/icons/...}}. Any file in any sub-folder can be used directly, or a specific image file can be given [https://askubuntu.com/questions/189231/where-are-the-stock-icon-names-defined-for-the-unity-panel-service-indicators-an/189262#189262]:
<source lang=bash>
notify-send -i face-laugh "I am happy"
notify-send -i task-due "Time to work dude!"
notify-send -i ~/myicon.png "This is my icon"
</source>

== Multimedia ==

=== paplay ===
<code>paplay</code> is the best alternative on "modern" linux

<source lang="bash">
<source lang="bash">
paplay /usr/share/games/xboard/sounds/woodthunk.wav
socat tcp:<host>:<port> - # <port> can be a port number or service name (telnet,imap...)
</source>
</source>

Some useful command-line options:
=== play ===
Play sound with ALSA. But often get ''under-run'' warnings (see [[Linux audio]]). Better use <code>paplay</code>.

<source lang="bash">
<source lang="bash">
play /usr/share/games/xboard/sounds/woodthunk.wav
socat -ly # Writes messages to syslog instead of stderr; severity as defined with option -d
play --single-threaded /usr/share/games/xboard/sounds/woodthunk.wav # Does not help
socat -v # Writes the transferred data to their target stream but also to stderr (text mode)
</source>
</source>


== Miscellaneous ==
== Python modules ==
Reference: https://til.simonwillison.net/python/stdlib-cli-tools
;[[http://htop.sourceforge.net/index.php htop]

:an improved ''top'' command
=== HTTP server ===
;iconv

:locale encoding conversions
<source lang="bash">
;konwert
python -m http.server
:fancy encoding conversions
</source>
;mimencode

:binary file conversion for the mail.;mkfifo
=== Base64 ===
:make FIFOs (named pipes)
<source lang="bash">
;pv, pipeview, pipebench
python3.11 -m base64 -h
:monitor the progress of data through a pipe
</source>
;reformime

:MIME E-mail reformatting tool
=== JSON ===
;stat
<source lang="bash">
:display file or file system status
echo '{"foo": "bar", "baz": [1, 2, 3]}' | python -m json.tool
;strace
</source>
:trace system calls and signals

;tee
=== Calendar ===
:read from standard input and write to standard output and files

;[http://mama.indstate.edu/users/ice/tree/ tree]
<source lang="bash">
:List the contents of directories in a tree-like format.
python -m calendar
;uuencode, uudecode
</source>
:binary file conversion for Unix.
;watch
:Execute a program periodically, showing output full screen
;[http://freshmeat.net/projects/xclip/ xclip]
:xclip is a command line interface to the X11 clipboard. It can also be used for copying files, as an alternative to sftp/scp, thus avoiding password prompts when X11 forwarding has already been setup. [http://gnuisance.net/blog/t/2007/xclip.html Check this guide].
;[http://www.vergenet.net/~conrad/software/xsel/ XSel]
:XSel is a command-line program for getting and setting the contents of the X selection (aka clipboard). Normally this is only accessible by manually highlighting information and pasting it with the middle mouse button.
;xselection
:Another package similar to ''xsel''. But I can't get it work correctly on OpenSUSE...

Latest revision as of 05:30, 29 April 2024

References

From The Linux Documentation Project:

File System

ack

ack is a grep-like program specifically for large source trees. It is distributed via package ack-grep. Add a symlink for convenience: 

sudo ln -s /usr/bin/ack-grep /usr/local/bin/ack

Features:

  • It's blazingly fast because it only searches the stuff you want searched.
  • ack ignores most of the crap you don't want to search
  • Lets you specify file types to search, as in --perl or --nohtml.
  • Color highlighting of search results.
  • Uses real Perl regular expressions, not a GNU subset.
# grep pattern $(find . -type f | grep -v '\.svn')
ack pattern                                           # Ack ignore CVS, SVN dir 
# grep pattern $(find . -name '*.pl' -or -name '*.pm' -or -name '*.pod' | grep -v .svn)
ack --perl pattern                                    # Specify file type to search
ack -f --perl > all-perl-files                        # Create rapidly list of files

Supported file types (more here): 

    --[no]asm          .asm .s
    --[no]batch        .bat .cmd
    --[no]binary       Binary files, as defined by Perl's -B op (default: off)
    --[no]cc           .c .h .xs
    --[no]cpp          .cpp .cc .cxx .m .hpp .hh .h .hxx
    --[no]css          .css
    --[no]hh           .h
    --[no]html         .htm .html .shtml .xhtml
    --[no]java         .java .properties
    --[no]js           .js
    --[no]make         Makefiles (including *.mk and *.mak)
    --[no]perl         .pl .pm .pod .t
    --[no]php          .php .phpt .php3 .php4 .php5 .phtml
    --[no]python       .py
    --[no]ruby         .rb .rhtml .rjs .rxml .erb .rake .spec
    --[no]shell        .sh .bash .csh .tcsh .ksh .zsh
    --[no]skipped      Files, but not directories, normally skipped by ack (default: off)
    --[no]text         Text files, as defined by Perl's -T op (default: off)
    --[no]vim          .vim
    --[no]xml          .xml .dtd .xsl .xslt .ent

ag

ag (home, github, ack.vim, ag.vim) is the silver searcher, and is actually much better than ack, which in turn is much better than grep ;-). Update: turns out that ripgrep is an even faster alternative.

On recent distribution, install from package: 

sudo apt install silversearcher-ag

To install from git (see README.md): 

sudo apt-get install software-properties-common
sudo apt-get install -y automake pkg-config libpcre3-dev zlib1g-dev liblzma-dev
git clone https://github.com/ggreer/the_silver_searcher.git
cd the_silver_searcher/
./build.sh
sudo make install
Tips
  • Use /b to find whole word matches only, or better -w:
ag "\bWHOLEWORD\b"
ag -w WHOLEWORD

chattr

chattr change file attributes on a Linux file system.

See man chattr for a list of attributes (immutable...)

cp

To copy content of a directory into another: 

cp -a /source/. /dest/

This copy recursively and preserve attributes -a, and copy the content with special syntax source/. with trailing dot.

To copy a directory recursively, create hard links instead of new copies (but creates new directories, not links): 

cp -lr <src> <target>

See #cpg for a patch that adds a progress bar.

cpg

cpg is a patched version of cp that adds a progress bar, advcpmv(github). 

cpg -ag src1 dst2              # Copy recursively with progress

Alternatives are to use rsync --progress (or rsync -P), or #gcp.

dd

Convert and copy a file.

dd returns current copy status when sent an HUP signal. Type in another shell: 

while :; sleep 5; do killall -s SIGHUP1 dd; done

This will force dd to update its status every 5sec.

Transfer speed is highly dependent on the block size! So make sure to use -bs=16M or something like that: 

 dd if=/dev/sda1 of=backup-sda1 bs=16M

Other recommended values for -bs are: 1MB [1], 64k [2] [3].

Note: to copy a whole file or device cat or pv are better alternatives (see Linux Disk Management). cat is at least faster than dd with default parameters [4], and pv looks for optimal block size to get highest speed. 

dd if=file1 of=file2             # slow
cat file1 > file2                # fast
dd if=file1 of=file2 bs=16M      # faster (maybe)
pv file1 > file2                 # fastest

Interesting links: 

# Create sparse file
dd of=sparse-file bs=1 count=0 seek=10           # SAME AS: truncate -s 10GB sparse-file
# Mount it
loop=`losetup --show -f sparse-file`
mkfs.ext4 $loop
mkdir myloop
mount $loop myloop

# Write myfile.iso to a USB drive
# !!! with -a, cp may try to create a new block device!
cp myfile.iso /dev/sdb

# Rip a cdrom to a .iso file
cat /dev/cdrom > myfile.iso

# Create a gzipped image
# !!! w/o redirection, gzip may skip non-regular file /dev/sdb
gzip -9 < /dev/sdb > /tmp/myimage.gz

entr

entr runs arbitrary commands when files change.

For instance: 

find -name *.c | entr make

fatsort

fatsort sorts directory structures of FAT filesystems. Many MP3 hardware players don't sort files automatically but play them in the order they were transmitted to the device. FATSort can help here. 

sudo fatsort /dev/sdb1

fd

fd is a simple, fast and user-friendly alternative to 'find'. 

fd
fd app
fd sh
fd sh --type f
fd -e md
fd -e md --exec wc -l
fd --hidden sample

find

Tip: Consider using fd as simpler and faster alternative.

See here for further examples on how to combine find and xargs.

FREQUENT PITFALLS!
  • ESCAPE THE SEMI-COLON — The semi-colon ; must be escaped in Bash !
  • USE - OR + WITH NUMERICAL VALUES — for instance -mtime 1 means exactly one day ago, whereas -mtime -1 means within the last day.
find . -name "*.jpg"                                     # find files/directories whose name matches specified pattern
find . -path "*/cur/*"                                   # find files/directories whose path spec matches specified pattern
find . -type d -name "cur"                               # find only directories whose name matches specified pattern
find . -exec echo '{}' \;                                # semi-colon must be escaped!
find . -exec echo one '{}' \; -exec echo two '{}' \;     # to execute several commands on one match
find . -name 'dirname' -prune        -o -iname '*.txt' -print
                                                         # Find all text files, but exclude any directory named dirname
find . -name 'dirname' -prune -false -o -iname '*.txt'   # ... idem (using -false, but no longer -print)
find . -path './path'  -prune -false -o -iname '*.txt'   # Find all text files, but exclude path ./path
find . -iname '*.zip' -exec sh -c 'unzip -l "$0" | egrep -o " [0-9]+ file"' '{}' \;
                                                         # How to use pipe in -exec.
find www -name '*.pdf' -printf %P\\n                     # Print PDF files in www/ but remove the leading www/ in output (%P)

Accelerating find: 

find / -mount -name "*splash*"                           # Only search on CURRENT file system (same as find -xdev)

Use also xargs to accelerate find: 

find . -name \*.jpg | xargs -r echo                      # much faster than find -exec
find . -name \*.jpg | xargs -rn 2 echo                   # limit #args passed to echo to 2 max.
find . -name \*.jpg -print0 | xargs -r0 ls               # Use NULL separator if files contains space or special char !!!

# find -exec command \; and xargs -i are equivalent
find . -name "*.txt" -exec echo '{}' is found. \;        # ONE at a time, but {} can be anywhere - and MULTIPLE times
find . -name "*.txt" -print0 | xargs -r0 -i echo '{}' is found.  # ... same, with xargs
# find -exec command {} + and xargs are equivalent
find . -name "*.txt" -exec echo The files are '{}' +     # FASTER - much like xargs. Only append AT THE END!
find . -name "*.txt" -print0 | xargs -r0 echo The files are   # ... same with xargs

# With mv, use -t DIRECTORY to give destination directory first
find . -size +100M -print0 | xargs -0 mv -t bigfiles/    # -t DIR inverts usual SOURCE... DESTINATION order
find . -size +100M -exec mv -t bigfiles/ {} +            # ... idem, without xargs

FSlint

FSlint is a utility to find and clean various forms of lint on a filesystem. I.E. unwanted or problematic cruft in your files or file names. For example, one form of lint it finds is duplicate files.

gcp

A variant of cp, but with progress bar: 

gcp -pR dir1 .

If complains about missing dbus: 

dbus-launch gcp -pR dir1 .

An alternative is to use a patched version of cp.

getfattr / setfattr

getfattr and setfattr gets/sets extended attributes of filesystem objects.

!!! These tools are really misleading, and many attributes are hidden. Usually you need to list the attributes explicitly !!! 

getfattr -d file                      # This SUPPOSEDLY returns *ALL* attributes, but actually *MANY* missing
getfattr -m - file                    # This MIGHT return more attributes
getfattr -n system.ntfs_attrib file   # View ntfs-3g special attributes
getfattr -d -e hex file               # View attribute values in HEX

grep / zgrep

grep -Rsl PATTERN [FILE]            # Recursive, no error output, only list filename
grep BASIC-REG-EXP-PATTERN [FILE]   # Use classic regexp (like "dma\|DMA")
egrep EXT-REG-EXP-PATTERN [FILE]    # Same as grep -E. Use extended regexp (like "dma|DMA")
fgrep FIXED-STRINGS-REG-EXP [FILE]  # Same as grep -F. Pattern is a list of strings to match.
grep -n PATTERN [FILE]              # Print matched line numbers.
grep -- "-s" [FILE]                 # Search for text "-s"
grep -e "-s" [FILE]                 # Search for text "-s" - alternative solution
grep -R -include=*.in PATTERN *     # Search recursively through folders, limiting to files matching pattern "*.in"
grep -R PATTERN *.in                # Idem, but matching pattern "*.in" also applies to folders.
grep -o PATTERN [FILE]              # (--only-matching) Print only the matched (non-empty) parts of a matching line.

zgrep works like grep, but also accepts compressed (.gz) files. 

zgrep PATTERN *.txt *.txt.gz        # Search in .txt file (compressed or not)

gzip / gunzip / zcat

gzip -c FILE                        # Compress FILE to stdout
gunzip -c FILE                      # Uncompress FiLe to stdout
zcat FILE                           # Equivalent to gunzip -c FILE

ls

Listing directory content: 

ls --full-time               # To list files full date & time
ls -d directory-name         # List directory entries, not content
                             #  eg. ls -d /etc/rc*
ls -lS | head                # List biggest files first - print only first 10 entries
ls -lt | head -n 20          # List most recent files first, print only 20 first entries

lsattr

lsattr list file attributes on a Linux second extended file system. 

sudo lsattr

---------------- ./memtest86+.elf
---------------- ./System.map-3.13.0-39-generic
---------------- ./vmlinuz-3.13.0-39-generic
---------------- ./initrd.img-3.13.0-39-generic
----------I----- ./grub
---------------- ./abi-3.13.0-39-generic
---------------- ./memtest86+_multiboot.bin
---------------- ./memtest86+.bin
---------------- ./config-3.13.0-39-generic
---------------- ./lost+found

ntfsundelete

ntfsundelete recover a delete file from an NTFS volume. Does not scan the disk.

See [5], [6].

par2

par2 is a very useful program for creating and using PAR2 files to detect damage in data files and repair them if necessary.

From the manpage, using an example 800MB test.mpg: 

par2 create test.mpg        # Will create *.par2 files, for a total of roughly 40MB (5% redundancy)
par2 verify test.mpg.par2   # to verify integrity
par2 repair test.mpg.par2   # to repair

*.par2 files stores error correction informations, in a number of recovery blocks. Repair is possible as long as there are enough recovery blocks available (on Usenet era, it was useful to have *.par2 files of increasing size such that to allow users to minimize bandwidth usage).

par2 is particularly useful to protect MP3 files on USB sticks (that have frequent FS failure over time): 

par2 create check.par2 *.mp3   # Generate recovery blocks (5% redundancy)
par2 verify check.par2         # Verify
par2 repair check.par2         # Repair

To increase redundancy: 

par2 create -r10 check.par2 *.mp3   # Generate recovery blocks (10% redundancy)

Note: to simulate file corruption, we can use: 

# Simulate 25kB corruption, from 10k-th byte.
dd if=/dev/zero of=output.mp3  bs=1024 count=25 seek=10 conv=notrunc

progress

progress [7] watches progress of other utilities like cp, mv, dd, tar. 

# display estimated I/O throughput and estimated remaining time for on going coreutils commands
progress -w       

# Start an heavy command and monitor it with -m and $!
tar czf images.tar.gz linuxmint-18-cinnamon-64bit.iso CentOS-7.0-1406-x86_64-DVD.iso CubLinux-1.0RC-amd64.iso | progress  -m  $!

pv

pv monitors the progress of data through a pipe. It can also be used to copy files or entire volume very rapidly.

For instance: 

pv </dev/sda >/dev/sdb

Also over the network using nc: 

# On Host A (receiver):
nc -l 2222 > /dev/sda
# On Host B (sender):
pv </dev/sda | nc hosta 2222                # 111MB avg throughput on direct link (cross-cable), SSD hard disks

rdfind

rdfind finds duplicate files, very rapidly. 

rdfind /foo/dir /bar/dir                       # Find duplicates in give dir and tell what could be gained
rdfind -makehardlinks true /foo/dir /bar/dir   # Remove dups by creating hardlinks

rg

rg or ripgrep is an extremelly fast alternative to grep (in fact even faster than ag the silver-searcher).

See ripgrep on GitHub for ripgrep.deb package.

rsync

Perfect copy over network (see this page for details) 

#If needed, pre-activate sudo on remote system. Flag -t required to solve 'sudo: no tty present and no askpass program specified'
#
# Also, this requires the following line in /etc/sudoers:
#
#    Defaults     !tty_tickets
#
stty -echo; ssh -t user@server sudo -v; stty echo

sudo rsync -aHAXS --delete --rsync-path "sudo rsync" --numeric-ids -h -v --exclude='lost+found' user@server:/remote/path  /local/path

More basic use of rsync: 

sudo rsync -aP src1 dst2           # Copy with progress

scalpel

scalpel recover files using a header / footer database (see [8]).

Install scalpel, then edit /etc/scalpel/scalpel.conf to select which file types to recover (based on header/footer). 

sudo apt-get install scalpel
sudo vi /etc/scalpel/scalpel.conf

Here we enable jpg and mp3 (see [9] for more): 

# GIF and JPG files (very common)
#	gif	y	5000000		\x47\x49\x46\x38\x37\x61	\x00\x3b
#  	gif	y 	5000000		\x47\x49\x46\x38\x39\x61	\x00\x00\x3b
 	jpg	y	200000000	\xff\xd8\xff\xe0\x00\x10	\xff\xd9
        jpg     y       200000000       \xff\xd8\xff\xe1                \xff\xd9 
# MP3
       mp3     y       8000000 \xFF\xFB??\x44\x00\x00
       mp3     y       8000000 \x57\x41\x56\45            \x00\x00\xFF\
       mp3     y       8000000 \xFF\xFB\xD0\            \xD1\x35\x51\xCC\
       mp3     y       8000000 \x49\x44\x33\
       mp3     y       8000000 \x4C\x41\x4D\x45\

Start scalpel: 

sudo scalpel /dev/sda1 -o output_dir      # output_dir must *not* be on disk being scanned


tar

tar archives and restores timestamps. When run as root, tar will also restore ownership and permissions. 

tar -cvzf archive.tgz somedir/                             # Archive & GZIP compress the directory somedir/ (recursively)
tar -cvjf archive.tar.bz2 somedir/                         # ... Use BZIP2 insteadl
tar -cvzf archive.tgz --exclude=notme somedir/             #  ... exclude any file matching name
tar -cvzf archive.tgz --exclude=somedir/not/me somedir/    #  ... exclude file at specific location

tar -xvzf archive.tgz                                      # Extract archive (will create somedir/)
tar -xvzf archive.tgz -C targetdir                         # ... to specified directory
tar -tvzf archive.tgz                                      # List the content (equiv. of tar --list -vzf ...)

tar cf - --sort=name --owner=root:0 --group=root:0 \
  --mtime='UTC 2019-01-01' . | gzip -n > invariant.tgz     # Create a stable archive w/o timestamps, etc 
                                                           # https://stackoverflow.com/questions/32997526/how-to-create-a-tar-file-that-omits-timestamps-for-its-contents
Troubleshootigng
  • On windows, getting errors like tar: ....myfile.txt: file changed as we read it.
No fix. We silent the error with the following script:
nice_tar()
{
    # Bloody tar returns a warning about modified dir on windows. Must silent the warning.
    tar "$@"
    RC=$?

    [ $RC -eq 0 -o $RC -eq 1 ] && return 0 || return $RC
}

testdisk / photorec

testdisk scans and repairs disk partitions. photorec recovers lost files from harddisk, digital camera and cdrom (including audio and photo files). 

photorec /log /d output_dir /dev/sda1

ug

ug or ugrep is an extremelly fast alternative to grep (in fact even faster than ag the silver-searcher, and comparable speed with ripgrep, but with some more powerful search capabilities).

See ugrep on GitHub (also nice instructions for setup on Vim).

Miscellaneous

badblocks
An utility to detect bad sectors on USB flash drive or so.
dosfslabel
Change FAT16/FAT32 disk label
sudo dosfslabel /dev/sdc1 UBUNTULIVE
mbr
Install MS-like Master Boot Record (see [10])
sudo apt-get install mbr
install-mbr -i n -p D -t 0 /dev/sda        # Fix MBR on /dev/sda
                                           # -i interrupt n = none (do not display a MBR prompt)
                                           # -p partition D = boot the partition with the bootable flag set
                                           # -t timeout 0 = do not wait to boot
dd if=/dev/sda of=opensource.mbr bs=512 count=1
install-mbr -i n -p D -t 0 opensource.mbr  # Create a file opensource.mbr containing the generated MBR
mkfs
Format disk utility
sudo mkfs.vfat -F 32 -c /dev/sdc1
ms-sys
Install MS-compatible Master Boot Record (similar to FIXMBR on WinXP rescue console, see [11])
# Get ms-sys from http://ms-sys.sourceforge.net/#Download
sudo apt-get install gettext
make
sudo make install
sudo ms-sys -m /dev/sdb
relocntfs
relocntfs (see [12]) is more or less the equivalent of the FIXBOOT command in the Windows Recovery Console. It is among other available on the Trinity Rescue Kit. For instance, to fix the boot record of a windows partition at /dev/sda2 issue the commands:
relocntfs /dev/sda2        
relocntfs -w /dev/sda2                    # To actually write the new boot record
stat
display file or file system status (like file access time, creation time, modification time...)
tree
List the contents of directories in a tree-like format.

Network

autossh

See Autossh.

dhclient

dhclient -r eth1        # Release DHCP lease on eth1
dhclient eth1           # Obtain fresh ip on eth1

dig

dig stands for domain information groper. It replaces the deprecated nslookup. It comes with package dnsutils: 

sudo apt-get install dnsutils

To get ip address of a given host: 

dig +short google.com | head -n 1      # In case host has multiple address

See also getent hosts or hostname.

To do a reverse DNS lookup:

  • Use option -x of dig command:
dig +short -x 82.78.227.176
# 176.176-191.227.78.82.in-addr.arpa.
# ivorde.ro.
  • Or query record PTR in in-addr.arpa domain:
dig +short ptr 176.227.78.82.in-addr.arpa.
# 176.176-191.227.78.82.in-addr.arpa.
# ivorde.ro.

getent

Use getent hosts to get ip address of a given host: 

getent hosts google.be
getent hosts google.be | awk '{ print $1; exit }'   # To get first ip address only

Note getent may succeed where dig fails (for instance host in /etc/hosts file or hosts with .local suffix).

host

Use host to do a reverse DNS lookup, i.e. get hostname from a given IP address: 

host 123.45.67.89

host can also do a regular DNS lookup and fetch alias names: 

host -t a www.facebook.com
# www.facebook.com is an alias for star.c10r.facebook.com.
# star.c10r.facebook.com has address 173.252.100.27

hostname

Use hostname -I to get local host ip address: 

hostname -I
hostname -I | awk '{ print $1 }'         # In case host has multiple interfaces / ip addresses

ifconfig

See Linux networking.

ip

See Linux networking.

iwconfig

See Wifi.

mail

mail — send and receive mail. 

# Using local SMTP
echo my message body here | /usr/bin/mail -s"My subject here" my_email@myserver.org

# Using ISP SMTP - no auth
echo "my message body here" | mail -S 'smtp=smtp.myisp.org:25' -S 'from=first.last@myisp.org' -s 'My subject here' my_email@myserver.org

multitee

multitee sends multiple inputs to multiple outputs.
Check this page.
Original is here, or can also be found here on Debian.

Here a patch to build it on Cygwin. The patch ports calls to BSD signal.h API (struct sigvec, sigvec(), sigmask(), sigblock()...) to the POSIX API (struct sigaction, sigaction(), sigprocmask(), sigfillset()...): 

$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin

Example of use: 

$ multitee 0-1,4,5 4> foo 5> bar             # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7                 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0   # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1                           # same as 'socat - FD:3'

netcat or nc

TCP-IP swiss army knife (equivalent of the telnet program. Check wikipedia:netcat. Also known as command nc).

See also Using Netcat (Linux Tips).

Using as telnet-like
nc www.immie.org 22                               # Connect to SSH port
nc www.immie.org ssh                              # idem, using port name
nc -v www.immie.org ssh                           # idem, verbose

Use nc also to test whether some ports are open, a bit like nmap. For instance: 

nc -vz www.immie.org 25                           # See if SMTP port is open
# www.immie.org [91.134.134.85] 25 (smtp) open
nc -vz www.immie.org 80                           # idem, with http port
Using TCP in multi-connect
Netcat will quit as soon as the first connection terminates. Use option -k to keep the connection alive:
# On DESTINATION host
nc -l -k 6666                                 # Keep connection alive
# On SOURCE host
echo foo | nc griffin 6666
echo bar | nc griffin 6666
Using UDP in multi-connect
Note that When nc is listening to a UDP socket, it 'locks on' to the source port and source IP of the first packet it receives. [13]. So one must give the source port (with option -p) on the SOURCE host as well, or second and next transmissions will be ignored.
# On DESTINATION host
nc -l -u 6666
# On SOURCE host
echo foo | nc -u -w 1 -p 6665 griffin 6666    # Enforce source port 6665. Option -w 1 to timeout after 1 sec
echo bar | nc -u -w 1 -p 6665 griffin 6666

netstat

See Linux networking.

nmap

nmap is the network exploration tool and security / port scanner. 

nmap localhost                    # Scan and print all open ports on 'localhost'
nmap -sP hostname                 # Simply test whether 'hostname' is accessible
nmap -sP 192.168.1.0/24           # Scan all local network with mask 192.168.1.0/24 (sudo: show mac address)
Detect host availability

One can use nmap in a bash script to test whether a given host is accessible, but it turns out that ping is faster: 

# Using nmap
nmap -sP $1 2>/dev/null | grep -q "1 host up"    # Too slow, does not always detect hosts

# ... or ping (faster)
if [ "$HOST_MACH" = "cygwin" ]; then
    ping $1 1 1 2>/dev/null | grep -q "1 packets received"
else
    ping -c 1 $1 2>/dev/null | grep -q "1 received"
fi
Detect all Raspberry pi on the network
sudo nmap -sP 192.168.1.0/24 | awk '/^Nmap/{ip=$NF}/B8:27:EB/{print ip}'
sudo nmap -sP 192.168.1.0/24 | grep -B2 Raspberry
Various scan
sudo nmap -A -T4 myserver.org         # Quick port scan on myserver.org
sudo nmap -A -T4 -p22 myserver.org    # Get SSH banner from server

nmcli

See Linux networking.

nm-tool

nm-tool is an utility to report NetworkManager state and devices. 

nm-tool                   # Print IP addresses, DNS servers, etc.

/etc/init.d/nscd

This is not really a command, but the init service Name Service Cache Daemon. 

sudo /etc/init.d/nscd restart                # To restart daemon and flush DNS cache

nslookup

nslookup is a program to query the DNS database. It is available on Linux and Windows, but on Linux is is deprecated by dig. 

nslookup 8.8.8.8        # Reverse DNS lookup

rpcinfo

rpcinfo reports RPC information 

rpcinfo -p              # Probe the portmapper on host, and print a list of all registered RPC programs

socat

Command-line utility that establishes two bidirectional byte streams and transfers data between them ([14]). socat is the more powerful version of netcat. Check the homepage. And also this page on Yobi for examples on how to use socat to bypass a proxy. 

# All these commands listen on localhost:53805, and forward to server:53806
socat TCP4-LISTEN:53805                TCP:server:53806      # Listen, and close as soon socket closes
socat TCP4-LISTEN:53805,reuseaddr,fork TCP:server:53806      # Idem, but fork so that listener continues after socket closes

socat -v ...                                                 # Send txfed data to stderr, with some formatting (text format)
socat -x ...                                                 # ... idem in hex format (can be combined with -v)
socat -ly                                                    # Writes messages to syslog instead of stderr; severity as defined with option -d

socat can easily tunnel connections: 

socat -ly 'TCP4-LISTEN:143,reuseaddr,fork' PROXY:ton.imap.server:143|TCP:134.27.168.36:8080

socat can also be used as SSH ProxyCommand: 

ProxyCommand socat -ly - 'PROXY:%h:%p|TCP:proxy.server:8080'
# Using socat v2.0.0 beta
ProxyCommand socat -ly - 'PROXY:%h:%p,proxyauth=user:pass|SSL,verify=0|PROXY:my.server:443,proxyauth=user:pass|TCP:big.brother.proxy:8080'

socat can be easily used as a replacement of telnet: 

socat tcp:<host>:<port> -       # <port> can be a port number or service name (telnet,imap...)

To establish UDP transfer, use: 

socat UDP-RECV:[port] STDOUT               # On destination host
socat STDIN UDP-DATAGRAM:[host]:[port]     # On source host

QUESTION TO MR DOEGOX:
- WHY -s in SED?
- Where does it get info on % for socat?

$ socat echo -
$ socat echo STDIO
$ socat echo STDOUT%STDIN
$ socat echo STDIO%STDIO
$ socat echo -%-
$ socat echo FD:1%FD:0
$ socat echo 1%0

/usr/local/bin/socat TCP:127.0.0.1:1234 -%EXEC:"awk 'BEGIN{print \"BANNER\";fflush()}/BANNER/{next}//{print;fflush()}'"

Testing socat:
- /usr/local/bin/socat - TCP-LISTEN:1234
- /usr/local/bin/socat TCP:127.0.0.1:1234 EXEC:"/bin/sed -s 's/foo/bar/g'"%EXEC:"/bin/sed -u 's/toto/tata/g'"

  First open a port for listening and connect it to STDIO
  In another shell, connect and do the black magic...


reference:
- http://eros.ph0k.eu/~nizox/twin/V2/modules/socat-1.6.0.0/EXAMPLES
- http://www.rschulz.eu/2008/09/ssh-proxycommand-without-netcat.html
		$ exec 3<>/dev/tcp/zeus.yobi.be/22;(cat <&3 & );cat >&3
		SSH-2.0-OpenSSH_5.1p1 Debian-3
- http://thesmithfam.org/blog/2006/05/23/bash-socket-programming-with-devtcp-2/
- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464
- http://www.mtu.net/~engstrom/ssh-proxy.php



ATTEMPT 3:
Window 1: socat -v -%'EXEC:/home/beq06659/.ssh/mys2.sh' TCP-LISTEN:1234
Window 2: /usr/local/bin/socat -ly -v -%"EXEC:sed -unf /home/beq06659/.ssh/sedy" "PROXY:ftp.noekeon.org:22|TCP:localhost:1234"

File mys2.sh:
	#! /bin/bash
	echo -e "HTTP/1.1 200 Connection established\r\n\r\n"
	cat
File sedy:
	1 s/^/SSH-2.0-OpenSSH_5.1\n/p; /SSH-2.0-OpenSSH_5.1/d; /SSH-2.0-OpenSSH_5.1/! p

LESSONS LEARNED:
- READ THE CHANGES DOC, especially for BETA SW
- SOCKET = bidir --> a filter = 4 descriptors
- PIPE = unidir --> a filter = 2 descriptor (stdin/stdout)

ss

See Linux networking.

wavemon

wavemon is a top-like monitoring tool for wifi connection.

To install: 

sudo apt install wavemon

To use: 

wavemon

wget

wget URL
wget URL --content-dispostion   # If download gets wrong name, use this option first
wget URL -O filename            # ... otherwise use this to rename download to 'filename'

whois

Find CIDR for www.facebook.com [15]: 

host -t a www.facebook.com
# www.facebook.com has address 69.171.228.40

whois 69.171.228.40 | grep CIDR
# CIDR:           69.171.224.0/19

Internet

httrack

httrack is a offline browser; it copies websites to a local directory. 

# Example: checking out progit boot
# -d                       stay on same principal domain
# -w                       mirror web sites
# -O /var/www              path for mirror/logfiles+cache
# -I0                      don't make an index
# -%v                      display on screen filesnames downloaded
# -P 127.0.0.1:8118        Proxy settings
# '+progit.org/figures/*'  +filter
# '-progit.org/book/*/*'   -filter
httrack http://progit.org/book/ -d -w -O /var/www -I0 -%v '+progit.org/figures/*' '-progit.org/book/*/*' [-P 127.0.0.1:8118]
sed -ri \"s_'http://progit.org/book'_'index.html'_\" /var/www/progit.org/book/*.html

# Example 2
# -W                       mirror web sites, semi-automatic (ask question)
httrack http://www.cplusplus.com/ -W -O /var/www -%v '-www.cplusplus.com/forum/*' '-www.cplusplus.com/src/*' '-www.cplusplus.com/member/*' [-P 127.0.0.1:8118]

# Example 3 - Use different path for cache/log files
#  -O /var/www,/tmp/httrack   path for mirror,path cache and logfiles
httrack http://www.argospress.com/Resources/systems-engineering/index.html -d -O /var/www,/tmp/httrack -I0 -%v [-P 127.0.0.1:8118]
sed -ri '/wms.assoc-amazon.com/{N; d}' /var/www/www.argospress.com/Resources/systems-engineering/*.htm

# Example 4 - Limit depth
#  -r2                     set mirror depth to 2 (i.e. given address + 1st links)
httrack http://powerman.name/doc/asciidoc -r2 -w -I0 [-P 127.0.0.1:8118]

Serial

stty

stty changes and prints terminal line settings. It usually comes pre-installed. 

stty -F /dev/ttyUSB0 -a         # Print all settings in human-readable form
stty -F /dev/ttyUSB0 --save     # Print all settings in stty-readable form

setserial

setserial get/set Linux serial port information.

See for instance [16] for usage.

Package Management

dpkg

Package manager for Debian / Ubuntu. See Package Management#Ubuntu / Debian

rpm

See Package Management#RPM's

System

dmidecode

dmidecode is a tool to decode DMI (SMBIOS) tables: 

sudo dmidecode

...
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
	Vendor: Dell Inc.
	Version: A19
	Release Date: 12/21/2009
	Address: 0xF0000
	Runtime Size: 64 kB
	ROM Size: 1728 kB
	Characteristics:
		ISA is supported
		PCI is supported
		PC Card (PCMCIA) is supported
		PNP is supported
		BIOS is upgradeable
		BIOS shadowing is allowed
...

inxi

inxi is yet another way to get information on hardware: 

inxi -Gx           # as current desktop user, don't use sudo!
# Graphics:  Card: Intel 3rd Gen Core processor Graphics Controller bus-ID: 00:02.0 
#            X.Org: 1.15.1 driver: intel Resolution: 1600x900@60.0hz 
#            GLX Renderer: Mesa DRI Intel Ivybridge Mobile GLX Version: 3.0 Mesa 10.1.0 Direct Rendering: Yes

loadkeys

loadkeys loads keyboard translation tables.

Use loadkeys to change the keyboard layout in console mode (i.e. Ctrl-Alt-F1 console). 

sudo loadkeys be                   # Load Belgian layout

lscpu

lscpu                              # Display information about the cpu architecture

lsmem

lsmem                              # List all ranges of available memory

lsmod / modinfo / modprobe

lsmod                              # Show the status of modules in the Linux Kernel
modinfo iwlagn                     # Show information about a Linux Kernel module
modprobe -r iwlagn                 # Remove module from the Linux Kernel
modprobe iwlagn 11n_disable=1      # Add module to the Linux Kernel, with specific parameters

lspci

lspci                              # List all PCI devices
lspci -v                           # List all PCI devices (verbose)

newgrp

newgrp log in to a new group 

 
newgrp - <group>       # Start a new session with <group> as primary group. - means to reinitialize environment.
umask 002              # Make sure all files / directories we create have group access

To run newgrp in a script, use the following construction [17]: 

newgrp groupb << END
    touch "myscript_output.txt"
END

Or better yet use the sg command instead.

pidinfo

Find the process ID of a running program 

pidof apache2                      # Find all pid of apache2

pgrep

Prints PID of the process matching given pattern. Never match pgrep itself! 

ps -fp $(pgrep autossh)            # Much better that "ps -aef | grep autossh", which match grep process itself

pkill

Send a signal (default TERM) to process matching given pattern. Never match pkill itself! 

pkill -0 autossh                   # Test if process "autossh" is running

ps

ps -ef                             # All processes            - full format

ps afl                             # Proc w ttys, all users   - long + forest
ps flx                             # Proc w/o ttys            - long + forest
ps aflx                            # All processes, all users - long + forest

ps au                              # Proc w ttys, all users   - user output
ps ux                              # Proc w/o ttys            - user output
ps aux                             # All processes, all users - user output

ps axs                             # All process, with signals

pv

pipe viewer — terminal-based tool for monitoring the progress of data through a pipeline 

pv -cN source < linux-2.2.20.tar.bz2 | bzcat | pv -CN bzcat | gzip -9 | pv -cN gzip > linux-2.2.20.tar.gz

   source: 4.48MB 0:02:01 [9.84kB/s] [========>                        ] 29% ETA 0:04:44
    bzcat:   26MB 0:02:01 [ 192kB/s] [               <=>                               ]
     gzip: 5.58MB 0:02:01 [  64kB/s] [                         <=>                     ]

savelog

savelog saves, rotates and compresses log files. This is similar to logrotate cron job. 

savelog -n ~/log/git-fetch.log

sg

sg executes command as different group ID. 

umask 002                     # Ensure that all files written will be group accessible
sg - <group> -c "mkdir foo"   # Run commands as group <group>

The following construction restart the current script under a different group (from [18]): 

if ! [ "${SBREADY:=false}" = true -o $(id -ng) = ${NEEDGRP:=wheel} ] ; then
    export SBREADY=true
    exec sg $NEEDGRP "$0" "$@"
fi

sudo / gksudo

See also Ubuntu's page at [linux reference script security sudo ubuntu]. 

ls | sudo tee /root/somefile                # Instead of redirection sudo ls > /root/somefile
ls | sudo tee -a /root/somefile             # ... same but append instead
sudo sh -c "ls > /root/somefile"            # Yet another way to do redirection with sudo
sudo !!                                     # Repeat last command with root privilege
sudo -i                                     # Get a root *login* shell
sudo su -                                   # ... another solution
sudo -s                                     # Root shell but keeping current environment
sudo su                                     # ... another solution
sudo -k                                     # Reset sudo password timeout

gksudo is the graphical equivalent of sudo: 

gksudo gedit /etc/fstab

showkey

showkey examine the codes sent by the keyboard (linux kernel only - for X keyboard events, see xev!). 

showkey -k                                  # Dump key codes (default)
showkey -s                                  # Dump scan codes

timeout / timelimit

timeout / timelimit are small binary to run a command with a given time limit. More exactly they send a given signal to the child process after some specified time. See also Perl - kill on ALARM signal). 

timeout -15 30 mplayer dvd://              # Show 30s of DVD movie, then kill mplayer with a TERM signal

Memory related

free
Display the amount of free and used memory (free -m to get info in MB)
free
free -m       # Show in megabyte
/proc/meminfo
All memory usage information
cat /proc/meminfo
vmstat
Summary of memory usage
vmstat

Miscellaneous

htop
an improved top command
lsb_release
lsb_release -a prints version information for the Linux release you're running.
Another option (also working on Debian), is to use cat /etc/*-release.
mkfifo
make FIFOs (named pipes)
pv, pipeview, pipebench
monitor the progress of data through a pipe
strace
trace system calls and signals
tee
read from standard input and write to standard output and files
uname
uname -a prints all system information, include machine name, kernel name & version.
watch
Execute a program periodically, showing output full screen
watch -n 0.5 myprog arg1 arg2          # Wait 0.5s between run
watch -cn 0.5 myprog arg1 arg2         # ... same with color
watch -cn 0.5 myprog \"arg w/ space\"  # BEWARE! arg must be "double" quoted
watch -cn 0.5 myprog ${*@Q}            # Quoted args, in case args are user defined in a script

User / Group Administration

groupadd / addgroup

Use groupadd, addgroup to create a new group. groupadd is the low-level utility. addgroup is the Debian more friendly version. 

sudo groupadd groupname                      # Create a new group groupname
sudo groupadd -g 123 groupname               # ... and specify gid of the new group
sudo addgroup groupname
sudo addgroup --gid 123 groupname

useradd / adduser

Use useradd to create new user. useradd is the low-level utility. adduser is the Debian more friendly version. 

sudo useradd username                        # Create a new user username
sudo useradd -s /bin/bash username           # ... specify the deafult shell in use for the user
sudo useradd -s /bin/bash -m username        # ... and create user home directory (from /etc/skel)
sudo useradd -u 1234 username                # Create a new user username with uid 1234
sudo useradd -g GRP1 -G GRP2,GRP3 username   # Specify primary and supplementary groups
sudo adduser username                        # Create a new user username
sudo adduser --system username               # Create a new system user (in the system uid range)
sudo adduser --shell /bin/bash username      # Use /bin/bash as default login shell

userdel / groupdel / deluser

Use userdel, groupdel, deluser to remove an existing user /group. deluser is the Debian more friendly version 

sudo userdel username                        # Remove existing user username
sudo groupdel groupname                      # Remove existing group groupname
sudo deluser username                        # Remove existing user username
sudo deluser --group groupname               # Remove existing group groupname

usermod

Use usermod to modify an existing user 

sudo usermod -g GRP1 username                # Modify the primary group of user username
sudo usermod -g GRP1 -G GRP2,GRP3 username   # ... and also the supplementary groups

gpasswd

Use gpasswd to administer /etc/group and /etc/gshadow 

sudo gpasswd -a $USER fuse                   # Add self to group 'fuse'

To make the new group effective: 

loginctl terminate-user $USERNAME            # This will FORCE LOGOUT !!!
su - $USERNAME                               # New shell with new groups

X

disper

TO install: 

sudo add-apt-repository ppa:wvengen/ppa
sudo apt-get update
sudo apt-get install

Some examples of use: 

disper -s               # Only enable primary screen (laptop screen)
disper -S               # Only enable secondary screen (external monitor)
disper -d auto -e       # Auto-detect + enable extended mode
disper -d auto -c       # clone mode

xbindkeys / xbindkeys-config

  • xbindkeys is a program that allows you to launch shell commands with your keyboard or your mouse under the X Window System. It links commands to keys or mouse buttons, using a configuration file. It's independent of the window manager and can capture all keyboard keys (ex: Power, Wake...).
  • xbindkeys-config is an easy to use gtk program for configuring Xbindkeys (see [19].

xclip

xclip is a command line interface to the X11 clipboard. It can also be used for copying files, as an alternative to sftp/scp, thus avoiding password prompts when X11 forwarding has already been setup. Check this guide.

xdpyinfo

Display information utility for X, like the dimension of the display (pixels vs millimeters) and the resolution (DPI): 

xdpyinfo | grep dimensions               # Return dimension of the desktop (size in pixels and millimeters)
xdpyinfo | grep resolution               # Return current screen resolution in DPI (dots per inch)

XSel / XSelection

XSel is a command-line program for getting and setting the contents of the X selection and context-menu clipboard. Normally the former is only accessible by manually highlighting information and pasting it with the middle mouse button, and the latter is only accessible by using context-menu cut/copy/paste command.

xselection is an equivalent package on OpenSUSE.

On Cygwin, one has to use getclip, putclip or device /dev/clipboard. 

xsel -p                                 # To get content of PRIMARY buffer (X selection)
xselection PRIMARY                      # (equivalent command for xselection)
xsel -b                                 # To get content of CLIPBOARD (context-menu clipboard)
xselection CLIPBOARD                    # (equivalent command for xselection)
echo sometext | xsel -p                 # To set PRIMARY buffer
echo sometext | xselection PRIMARY -    # (equivalent command for xselection)
xselection PRIMARY sometext             # (equivalent command for xselection)
echo sometext | xsel -b                 # To set CLIPBOARD buffer
echo sometext | xselection CLIPBOARD -  # (equivalent command for xselection)
xselection CLIPBOARD sometext           # (equivalent command for xselection)

xvkbd

xvkbd is a virtual (graphical) keyboard program for X Window System which provides facility to enter characters onto other clients (softwares) by clicking on a keyboard displayed on the screen. It can also be used to send keyboard events through command-line: 

xvkbd -xsendevent -text text-string

xwininfo

xwininfo is the window information utility for X. Could be used to find the geometry of a window. See here 

xwininfo                               # Return information on the clicked windows
xwininfo -root                         # ... or on the root window
xwininfo -root |awk '/Width/{print $2} # Get Width of the current desktop

Text Manipulation

column

Format input into multiple columns 

cat | column -l
123 456
98765 98765
^D
# 123   456
# 98765 98765

csvquote

csvquote is a nice wrapper tool to pre-/post-process CSV files to ease processing by tools like AWK (to handle comma within quotes) 

csvquote foobar.csv | cut -d ',' -f 5 | sort | uniq -c | csvquote -u
cat foobar.csv | csvquote | cut -d ',' -f 7,4,2 | csvquote -u
csvquote -t foobar.tsv | wc -l
csvquote -q "'" foobar.csv | sort -t, -k3 | csvquote -u
csvquote foobar.csv | awk -F, '{sum+=$3} END {print sum}'

cut

TBC

paste

TBC

fold

fold -w 40 FILE              # Wrap FILE to 40-char width

expand

expand -t 4 FILE             # Convert tabs to spaces (4)
expand -t 4,10 FILE          # Set explicit tab positions

fmt

TBC

less

command | less -S                      # To wrap (aka fold) long-lines (or use '-','S' within less)

nl

nl numbers lines of files 

nl -ba FILE                  # Number *all* lines of FILE

rl, shuf

rl is a command-line tool that reads lines from an input file or stdin, randomizes the lines and outputs a specified number of lines (requires package randomize-lines).

shuf generates random permutations.


They are alternative to sort -R (note that the latter does not work with duplicate lines).

sed

Moved to page dedicated to Sed.

sort

sort is the well-know text sort utility. 

sort -R shuffle.txt                           # shuffle the lines (use random hash) - fail with duplicate lines
nl -ba file | sort -R | sed 's/^.*[0-9]\t//'  # ... idem, but workaround for dup lines

sort by default takes into account the current locale. To get the traditional sort order that uses native byte values: 

LC_ALL=C sort myfile.txt                      # traditional sort, indep of current locale

Use -o file to sort in-place, or use the sponge utility: 

sort file -o file                             # sort in-place
sort file | sponge file                       # soak sorted file and write it back

Use LC_ALL=C for huge speed improvement. Also increasing the internal buffer helps: 

LC_ALL=C sort -S 2G file

tr

tr translates or deletes characters.

Some basic uses: 

echo lowercase | tr a-z A-Z                # turns lowercase into uppercase
echo lowercase | tr [:lower:] [:upper:]    # idem
echo encrypt me please | tr a-z n-za-m     # a very simple rot13 implementation

A powerful rot-n implementation [20]: 

dual=abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
phrase='hello there'
n=13
newphrase=$(echo $phrase | tr ${dual:0:26} ${dual:$n:26})
echo ${newphrase}

tr can also be used to split words into newlines: 

tr ' ' '\n' file

figlet, toilet... (fun)

Some funny banner-like tool:

  • figlet [21]
  • toilet [22]
  • cowsay
  • ponysay
  • lolcat

Image manipulation

netpbm

Netpbm is a package of graphics programs and programming libraries. There are over 220 separate programs in the package, most of which have pbm, pgm, ppm, or pnm in their names. For example, pnmscale and giftopnm.

For example, you might use pnmscale to shrink an image by 10%. Or use pnmcomp to overlay one image on top of another. Or use pbmtext to create an image of text. Or reduce the number of colors in an image with pnmquant. 

anytopnm myfile.gif | pnmtopng > myfile.png                    # Convert a GIF file to PNG
anytopnm oll28.gif | pnmflip -r90 | ppmtogif > oll28-90.gif    # Rotate and output GIF

JPEG file manipulation can be done with djpeg and cjpeg from package libjpeg-turbo-progs: 

djpeg foo.jpg | pnmflip -r90 | cjpeg)    # Lossy rotate of jpeg

Development

addr2line

addr2Line converts addresses into file names and line numbers.

ar

ar creates, modifies, and extracts from archives. It can be used to generate static libraries (.a files) for use with gcc.

diff

Use diff to compare 2 files/directories together and/or to generate patch files. Use colordiff (package colordiff) to have colored output.! 

diff    old new                        # View difference (default ed script format)
diff -u old new >new.patch             # Generate a patch file (universal format)
diff -y old new                        # View difference side-by-side (as with sdiff)

#Color with DIFF:
colordiff -u old new                   # colordiff can also colorize a diff output package colordiff
diff old new | colordiff               # ... idem

#Color with WDIFF:
wdiff -n old new | colordiff           # But need option '-n'

#Piping to LESS with COLORS
diff -u old new | colordiff | less -R  # To keep color with pager

#DIFF / WDIFF combo
diff --strip-trailing-cr -bu old new | wdiff -d | colordiff | less -R             # Handy when wdiff miss some options...

Patch can be applied with the command patch (see below). Be careful regarding file names / directory structures when generating patch files! Sometimes you'll need to edit the generated patch files to fix the directory structure.

ldd

ldd print shared object dependencies.

See also readelf and lddtree (package pax-utils). 

ldd /bin/ls
#       linux-vdso.so.1 (0x00007ffcc3563000)
#       libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f87e5459000)
#       libcap.so.2 => /lib64/libcap.so.2 (0x00007f87e5254000)
#       libc.so.6 => /lib64/libc.so.6 (0x00007f87e4e92000)
#       libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f87e4c22000)
#       libdl.so.2 => /lib64/libdl.so.2 (0x00007f87e4a1e000)
#       /lib64/ld-linux-x86-64.so.2 (0x00005574bf12e000)
#       libattr.so.1 => /lib64/libattr.so.1 (0x00007f87e4817000)
#       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f87e45fa000)

lddtree

lddtree


readelf

readelf displays information on ELF files. 

# View libraries linked in .elf (but only directly linked, these may yet load other libs)
readelf -d obs-ios-camera-source.so | grep NEEDED
#  0x0000000000000001 (NEEDED)             Shared library: [libobs.so.0]
#  0x0000000000000001 (NEEDED)             Shared library: [libavcodec.so.58]
#  ...

patch

Use patch to apply a patch previously generated with the command diff. patch use is pretty much straightforward, one mainly has to pay attention to directory structure / blanks.

Complete example using diff: 

$ cp project project-patched     
$ cd project-patched
$ vi somefile                                             # We start modifying the copy
$ cd ..
$ diff -u project project-patched >project.patch          # We generate the patch file (universal format)
$ cd project                                              # We cd into project directory to patch
$ patch -lNp1<../project.patch                            # We apply the patch in place
  • -l — ignore white spaces (very handy when copying patch from internet page for instance).
  • -N — ignore patch that seems reversed or already applied (same as --forward).
  • -p1 — tells patch to ignore the first (i.e. one level of) directory in the file name given in the patch header.

The value to use for option p depends actually on the patch header:

  • First example - same root directory for both to and from file
    • --- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2/jpg.c       2009-08-25 16:06:05.242378300 +0200

    $ patch -lNp0 <project.patch                        # First example
  • second example - different root directory between to and from file
    • --- outguess-0.2/jpg.c       2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2-patched/jpg.c       2009-08-25 16:06:05.242378300 +0200

    $ cd project_dir; patch -lNp1 <../project.patch     # Second example

vbindiff

vbindiff — hexadecimal file display and comparison 

vbindiff file1.bin file2.bin               # Compare and view 2 binary files

Scripting

Some commands very useful when writing scripts

command

From the manpage, command execute a simple command... 

GDB=$(command -v gdb)                  # Get path to gdb

command -v behaves much like which, except that on Bash:

  • command is a shell built-in.
  • it treats aliases differently.

Miscellaneous

base64

base64 from coreutils 

echo QWxhZGRpbjpvcGVuIHNlc2FtZQ== | base64 --decode
echo `echo QWxhZGRpbjpvcGVuIHNlc2FtZQ== | base64 --decode`


cksum / jacksum / md5sum / sha1sum / sum

cksum file                                              # Generate a CRC similar to CRC32, but in decimal, and with length appended.
cksum -o3 file                                          # Generate a CRC32 checksum (BSD only)
sum file                                                # Generate a checksum
md5sum file                                             # Hash file using MD5
sha1sum file                                            # Hash file using SHA1
jacksum -a crc32 file                                   # Hash using CRC32 (dec)
jacksum -a crc32 -E hexup file                          # Hash using CRC32 (hex)

date

date +'%F %T'                # YYYY-mm-dd HH:MM:SS
date +'%Y%m%d%H%M%S"         # YYYmmddHHMMSS
date +'%s'                   # Get current epoch time (seconds since 1970-01-01 00:00:00 UTC)
date --date='@1498485311'    # Convert epoch to human readable time

direnv

direnv is an extension for your shell. It augments existing shells with a new feature that can load and unload environment variables depending on the current directory. 

# Create a new folder for demo purposes.
$ mkdir ~/my-project
$ cd ~/my-project

# Show that the FOO environment variable is not loaded.
$ echo ${FOO-nope}
nope

# Create a new .envrc. This file is bash code that is going to be loaded by
# direnv.
$ echo export FOO=foo > .envrc
.envrc is not allowed

# The security mechanism didn't allow to load the .envrc. Since we trust it,
# let's allow its execution.
$ direnv allow .
direnv: reloading
direnv: loading .envrc
direnv export: +FOO

# Show that the FOO environment variable is loaded.
$ echo ${FOO-nope}
foo

# Exit the project
$ cd ..
direnv: unloading

# And now FOO is unset again
$ echo ${FOO-nope}
nope

dog

dog is better than cat. It writes the contents of each give file, URL, or the standard input to standard output.

dtach

dtach simple program that emulates the detach feature of screen. A session in dtach is a single instance in which a program is running under the control of dtach. The program is disassociated from the original terminal, and is thus protected from your original terminal being disconnected for some reason.

echo

echo -e "Some text\n...on 2 lines..."                    # Enable interpretation of backslash escapes (must be quoted!)

expect

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.

hyperfine

hyperfine is a command-line benchmarking tool (benchmark tool). 

hyperfine --warmup 3 'fd -HI "[0-9]\.jpg"' 'find -name "*[0-9].jpg"'

Hyperfine may produce results in CSV, JSON, Markdown.

iconv

locale encoding conversions

konwert

fancy encoding conversions

mc (Midnight Commander)

mc or Midnight Commander is a powerful file manager working in a shell terminal.
See dedicated page on this wiki.

mimencode

binary file conversion for the mail.

munpack

Use munpack to extract MIME attachment from email [23]: 

cat Message
# ...
# 
# --PGP_Universal_31BF743E_DCBDB54C_D4397473_15481C77
# Content-Type: application/octet-stream;
# 	name="tuleap_ex.txt.gpg"
# Content-Transfer-Encoding: BASE64
# Content-Disposition: attachment;
# 	filename="tuleap_ex.txt.gpg"
# 
# hQIOA1uCgmTNqsbUEAgAqbp/CXJfNQmZu8RHtZaP6Fm5gi+sjyothO19faktgOFrVTsame5ZW8yi
# ...
# ...
# NnRRvRI+TcohkYsisQy4oiUA
# --PGP_Universal_31BF743E_DCBDB54C_D4397473_15481C77--
# 
sudo apt install mpack
munpack -f Message

mrxvt

A multi-tab version of rxvt (terminal-emulator).

parallel

parallel runs many job in parallel. 

parallel -i chronic sh -c "date; echo {}"

Note that this is parallel from moreutils. There is also a separate package parallel with different options.

recode

recode /cl../cr <dos.txt >mac.txt
recode /cr.. <mac.txt >unix.txt
recode ../cl <unix.txt >dos.txt
recode charset1/surface1..charset2/surface2 <input.txt >output.txt
recode /QP.. <qp.txt >normal.txt                                    # To convert quoted-printable text
charset surface
us ASCII (7 bits) /cr Carriage return as end of line (Mac text)
l1 ISO Latin-1 (ISO-8859-1, Western Europe, 8 bits) /cl Carriage return line feed as end of line (DOS text)
EUCJP EUC-JP for Japanese (Unix) / Line feed as end of line (Unix text)
SJIS Shift-JIS for Japanese (Microsoft) /d1 Human readable bytewise decimal dump
ISO2022JP Mail encoding for Japanese (7 bits) /x1 Human readable bytewise hexidecimal dump
u2 UCS-2 (Universal Character Set, 2 bytes) /64 Base64 encoded text (see also base64) from coreutils.
u8 UTF-8 (Universal Transformation Format, 8 bits) /QP Quoted-Printable encoded text

reformime

MIME E-mail reformatting tool

screen

Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.

See also tmux for a better alternatives that supports splitting in panes, mousewheel support...

Launch with 

screen                                      # Start a new screen session
screen -r                                   # Resumes a detacked screen session
screen -d -r                                # Reattach a session and if necessary detach it first
screen -d -R                                # Reattach a session and if necessary detach or even create it first
screen -D -r                                # Reattach a session. If necessary detach and logout remotely first
screen -D -R                                # Attach here and now, ie:
                                            # - If no session is running, create it
                                            # - Otherwise reattach the session. If necessary detach/logout remotely first

For convenience one may add the following alias to .bashrc: 

alias screen=screen -D -R

Useful shortcuts:

key description
C-a c Open a new instance
C-a Space Switch to next instance
C-a d Detach from screen. Reattach later with screen -r
C-a D D Power detach, i.e. detach and logout

byobu is a wrapper around screen, which makes it prettier and more powerful (see this also).

Use reptyr to capture an already running process and attach it to a new terminal. Very handy if you start a process, and you regret not having started screen first...

sponge

sponge (package moreutils) soaks up standard input and write to a file. It can be used to easily edit file in-place: 

sed -r '...' FILE | grep ... | sponge FILE                   # Pipeline from and to same file!

tabs

Set tabulation size of the terminal. 

tabs 4               # Set default tab size to 4
tabs 8               # Set default tab size to 8 (the default)

Note that some applications are not impacted by this setting or have alternate solution: 

less -x1,5                                       # LESS - sets tabs
git config --global core.pager 'less -x1,5'      # GIT
diff -t --tabsize=4                              # DIFF - alternate solution

tesseract

A tool to perform OCR.

Example of script AskUbuntu: 

#!/bin/bash

SCR_IMG=".screentemp.png"
TEMP_TXT=".screentext.txt"

gnome-screenshot -a --file=$SCR_IMG
tesseract $SCR_IMG $TEMP_TXT -l eng
cat $TEMP_TXT* | xsel -b

rm $SCR_IMG $TEMP_TXT*

timeout

timeout runs a command with a time limit.

On MinGW, timeout is not available. Instead there is a bash-script equivalent.

tmux

See page Tmux.

uuencode, uudecode

Binary file conversion for Unix.

wc

Counts lines, words and character in a file

xargs

For more examples, see combinations with the command find 

#By default, xargs cuts at WHITE SPACE and NEWLINE. Use -d '\n' to only cut at NEWLINES
for i in $(locate .pdf); do basename $i; done      # WRONG
locate .pdf | xargs -rd '\n' -n 1 basemane          # Correct

Hex Tools

hd / hexdump

ASCII, decimal, hexadecimal, octal dump. 

hexdump -e '"%2x"' <myfile>             # Convert myfile into a long hexadecimal string - ! See DOUBLE-QUOTED parameter
hexdump -C <myfile>                     # Canonical hex + ascii display (16 bytes)
hd <myfile>                             # (idem)

See also od and xxd. There are also hex editors, like hexer and hexedit (adapts to width of the terminal!).

od

od - dump files in octal and other formats 

od -xc file                                              # Output file in ascii and hex output (2-byte, ! little-endian)
od -t x1 -a --width=32 testvar.sh                        # Output 32-byte at once

See also hd / hexdump and xxd.

xxd

Make a hexdump or do the reverse. This tool is delivered along with vim. Probably the BEST out there 

xxd -g8 -c32 <file>                        # Output 32 bytes, grouped in 8-byte columns
xxd -p -c64 <file>                         # Output 64 bytes per line, plain (postscript) mode
echo 202122 | xxd -r -p                    # Convert hexdump to a binary string
echo -n ' !"' | xxd -p                     # Convert binary string to hexdump - DON'T FORGET -n
echo -n ' !"' | xxd -u -p                  # Convert binary string to hexdump - DON'T FORGET -n (uppercase)

See also hd / hexdump and od.

xxd can be easily used in Vim to hexedit a file: 

" To convert current buffer to hex
:%xxd
" To convert current buffer from hex
:%xxd -r

Security

openssl rand

openssl rand generates random strings: 

openssl rand -hex 16                      # Generate a 16-byte random string in hexadecimal
openssl rand -base64 16                   # Generate a 16-byte random string in base64

Desktop

notify-send

notify-send is a program to send desktop notifications. 

notify-send "summary" "body message"

If messages are not visible, use either -t 0 to remove time limit, or critical urgency [24]: 

 
notify-send -u critical "Urgent" "This is urgent!"
notify-send -t 0 "Nag" "This is a nag screen."

notify-send uses icons from /usr/share/icons/gnome/32x32 and /usr/share/notify-osd/icons/.... Any file in any sub-folder can be used directly, or a specific image file can be given [25]: 

notify-send -i face-laugh "I am happy"
notify-send -i task-due "Time to work dude!"
notify-send -i ~/myicon.png "This is my icon"

Multimedia

paplay

paplay is the best alternative on "modern" linux 

paplay /usr/share/games/xboard/sounds/woodthunk.wav

play

Play sound with ALSA. But often get under-run warnings (see Linux audio). Better use paplay. 

play /usr/share/games/xboard/sounds/woodthunk.wav
play --single-threaded /usr/share/games/xboard/sounds/woodthunk.wav    # Does not help

Python modules

Reference: https://til.simonwillison.net/python/stdlib-cli-tools

HTTP server

python -m http.server

Base64

python3.11 -m base64 -h

JSON

echo '{"foo": "bar", "baz": [1, 2, 3]}' | python -m json.tool

Calendar

python -m calendar
Retrieved from "https://miki.immie.org/mediawiki/index.php?title=Linux_Commands&oldid=17539"