Linux Commands: Difference between revisions
(→sort) |
(→sponge) |
||
| Line 922: | Line 922: | ||
sort file -o file # sort in-place |
sort file -o file # sort in-place |
||
sort file | sponge file # soak sorted file and write it back |
sort file | sponge file # soak sorted file and write it back |
||
</source> |
|||
=== sponge === |
|||
'''sponge''' soaks up standard input and write to a file |
|||
<source lang=bash> |
|||
sed '...' file | grep '...' | sponge file # pipeline from and to same file! |
|||
</source> |
</source> |
||
Revision as of 17:30, 13 November 2013
File System
ack
ack is a grep-like program specifically for large source trees. It is distributed via package ack-grep. Add a symlink for convenience:
sudo ln -s /usr/bin/ack-grep /usr/local/bin/ack
Features:
- It's blazingly fast because it only searches the stuff you want searched.
- ack ignores most of the crap you don't want to search
- Lets you specify file types to search, as in --perl or --nohtml.
- Color highlighting of search results.
- Uses real Perl regular expressions, not a GNU subset.
# grep pattern $(find . -type f | grep -v '\.svn')
ack pattern # Ack ignore CVS, SVN dir
# grep pattern $(find . -name '*.pl' -or -name '*.pm' -or -name '*.pod' | grep -v .svn)
ack --perl pattern # Specify file type to search
ack -f --perl > all-perl-files # Create rapidly list of files
Supported file types (more here):
--[no]asm .asm .s
--[no]batch .bat .cmd
--[no]binary Binary files, as defined by Perl's -B op (default: off)
--[no]cc .c .h .xs
--[no]cpp .cpp .cc .cxx .m .hpp .hh .h .hxx
--[no]css .css
--[no]hh .h
--[no]html .htm .html .shtml .xhtml
--[no]java .java .properties
--[no]js .js
--[no]make Makefiles (including *.mk and *.mak)
--[no]perl .pl .pm .pod .t
--[no]php .php .phpt .php3 .php4 .php5 .phtml
--[no]python .py
--[no]ruby .rb .rhtml .rjs .rxml .erb .rake .spec
--[no]shell .sh .bash .csh .tcsh .ksh .zsh
--[no]skipped Files, but not directories, normally skipped by ack (default: off)
--[no]text Text files, as defined by Perl's -T op (default: off)
--[no]vim .vim
--[no]xml .xml .dtd .xsl .xslt .ent
dd
Convert and copy a file.
dd returns current copy status when sent an HUP signal. Type in another shell:
while :; sleep 5; do killall -s SIGHUP1 dd; done
This will force dd to update its status every 5sec.
find
See here for further examples on how to combine find and xargs.
Frequent pitfall! The semi-colon ; must be escaped in Bash !
find . -name "*.jpg" # find files/directories whose name matches specified pattern
find . -path "*/cur/*" # find files/directories whose path spec matches specified pattern
find . -type d -name "cur" # find only directories whose name matches specified pattern
find . -exec echo '{}' \; # semi-colon must be escaped!
find . -exec echo one '{}' \; -exec echo two '{}' \; # to execute several commands on one match
find . -name \*.jpg | xargs echo # much faster version using xargs
find . -name \*.jpg | xargs -n 2 echo # limit #args passed to echo to 2 max.
find . -name \*.jpg -print0 | xargs -0 ls # Use NULL separator if files contains space or special char !!!
find . -name 'dirname' -prune -o -iname '*.txt' -print
# Find all text files, but exclude any directory named dirname
find . -name 'dirname' -prune -false -o -iname '*.txt' # ... idem (using -false, but no longer -print)
find . -path './path' -prune -false -o -iname '*.txt' # Find all text files, but exclude path ./path
find . -iname '*.zip' -exec sh -c 'unzip -l "$0" | egrep -o " [0-9]+ file"' '{}' \;
# How to use pipe in -exec.
find www -name '*.pdf' -printf %P\\n # Print PDF files in www/ but remove the leading www/ in output (%P)
Accelerating find:
find / -mount -name "*splash*" # Only search on CURRENT file system (same as find -xdev)
With mv, use -t DIRECTORY to give destination directory first:
find . -size +100M -print0 | xargs -0 mv -t bigfiles/ # -t DIR inverts usual SOURCE... DESTINATION order
find . -size +100M -exec mv -t bigfiles/ {} + # ... idem, without xargs
Equivalence between -exec command \; and xargs -i, and -exec command {} + and xargs:
find . -name "*.txt" -exec echo '{}' is found. \; # ONE at a time, but {} can be anywhere - and MULTIPLE times
find . -name "*.txt" -print0 | xargs -0 -i echo '{}' is found. # ... same, with xargs
find . -name "*.txt" -exec echo The files are '{}' + # FASTER - much like xargs. Only append AT THE END!
find . -name "*.txt" -print0 | xargs -0 echo The files are # ... same with xargs
FSlint
FSlint is a utility to find and clean various forms of lint on a filesystem. I.E. unwanted or problematic cruft in your files or file names. For example, one form of lint it finds is duplicate files.
getfattr / setfattr
getfattr and setfattr gets/sets extended attributes of filesystem objects.
!!! These tools are really misleading, and many attributes are hidden. Usually you need to list the attributes explicitly !!!
getfattr -d file # This SUPPOSEDLY returns *ALL* attributes, but actually *MANY* missing
getfattr -m - file # This MIGHT return more attributes
getfattr -n system.ntfs_attrib file # View ntfs-3g special attributes
getfattr -d -e hex file # View attribute values in HEX
grep / zgrep
grep -Rsl PATTERN [FILE] # Recursive, no error output, only list filename
grep BASIC-REG-EXP-PATTERN [FILE] # Use classic regexp (like "dma\|DMA")
egrep EXT-REG-EXP-PATTERN [FILE] # Same as grep -E. Use extended regexp (like "dma|DMA")
fgrep FIXED-STRINGS-REG-EXP [FILE] # Same as grep -F. Pattern is a list of strings to match.
grep -n PATTERN [FILE] # Print matched line numbers.
grep -- "-s" [FILE] # Search for text "-s"
grep -e "-s" [FILE] # Search for text "-s" - alternative solution
grep -R -include=*.in PATTERN * # Search recursively through folders, limiting to files matching pattern "*.in"
grep -R PATTERN *.in # Idem, but matching pattern "*.in" also applies to folders.
grep -o PATTERN [FILE] # (--only-matching) Print only the matched (non-empty) parts of a matching line.
zgrep works like grep, but also accepts compressed (.gz) files.
zgrep PATTERN *.txt *.txt.gz # Search in .txt file (compressed or not)
gzip / gunzip / zcat
gzip -c FILE # Compress FILE to stdout
gunzip -c FILE # Uncompress FiLe to stdout
zcat FILE # Equivalent to gunzip -c FILE
ls
Listing directory content:
ls --full-time # To list files full date & time
ls -d directory-name # List directory entries, not content
# eg. ls -d /etc/rc*
ls -lS | head # List biggest files first - print only first 10 entries
ls -lt | head -n 20 # List most recent files first, print only 20 first entries
ntfsundelete
ntfsundelete recover a delete file from an NTFS volume. Does not scan the disk.
scalpel
scalpel recover files using a header / footer database (see [3]).
Install scalpel, then edit /etc/scalpel/scalpel.conf to select which file types to recover (based on header/footer).
sudo apt-get install scalpel
sudo vi /etc/scalpel/scalpel.conf
Here we enable jpg and mp3 (see [4] for more):
# GIF and JPG files (very common)
# gif y 5000000 \x47\x49\x46\x38\x37\x61 \x00\x3b
# gif y 5000000 \x47\x49\x46\x38\x39\x61 \x00\x00\x3b
jpg y 200000000 \xff\xd8\xff\xe0\x00\x10 \xff\xd9
jpg y 200000000 \xff\xd8\xff\xe1 \xff\xd9
# MP3
mp3 y 8000000 \xFF\xFB??\x44\x00\x00
mp3 y 8000000 \x57\x41\x56\45 \x00\x00\xFF\
mp3 y 8000000 \xFF\xFB\xD0\ \xD1\x35\x51\xCC\
mp3 y 8000000 \x49\x44\x33\
mp3 y 8000000 \x4C\x41\x4D\x45\
Start scalpel:
sudo scalpel /dev/sda1 -o output_dir # output_dir must *not* be on disk being scanned
tar
tar archives and restores timestamps. When run as root, tar will also restore ownership and permissions.
tar -cvzf archive.tgz somedir/ # Archive & GZIP compress the directory somedir/ (recursively)
tar -cvjf archive.tar.bz2 somedir/ # ... Use BZIP2 insteadl
tar -cvzf archive.tgz --exclude=notme somedir/ # ... exclude any file matching name
tar -cvzf archive.tgz --exclude=somedir/not/me somedir/ # ... exclude file at specific location
tar -xvzf archive.tgz # Extract archive (will create somedir/)
tar -xvzf archive.tgz -C targetdir # ... to specified directory
tar -tvzf archive.tgz # List the content (equiv. of tar --list -vzf ...)
testdisk / photorec
testdisk scans and repairs disk partitions. photorec recovers lost files from harddisk, digital camera and cdrom (including audio and photo files).
photorec /log /d output_dir /dev/sda1
Miscellaneous
- badblocks
- An utility to detect bad sectors on USB flash drive or so.
- dosfslabel
- Change FAT16/FAT32 disk label
sudo dosfslabel /dev/sdc1 UBUNTULIVE
- install-mbr
- Install MS-like Master Boot Record (see [5])
sudo apt-get install mbr
install-mbr -i n -p D -t 0 /dev/sda # Fix MBR on /dev/sda
# -i interrupt n = none (do not display a MBR prompt)
# -p partition D = boot the partition with the bootable flag set
# -t timeout 0 = do not wait to boot
dd if=/dev/sda of=opensource.mbr bs=512 count=1
install-mbr -i n -p D -t 0 opensource.mbr # Create a file opensource.mbr containing the generated MBR
- mkfs
- Format disk utility
sudo mkfs.vfat -F 32 -c /dev/sdc1
- ms-sys
- Install MS-compatible Master Boot Record (similar to FIXMBR on WinXP rescue console, see [6])
# Get ms-sys from http://ms-sys.sourceforge.net/#Download
sudo apt-get install gettext
make
sudo make install
sudo ms-sys -m /dev/sdb
- relocntfs
- relocntfs (see [7]) is more or less the equivalent of the FIXBOOT command in the Windows Recovery Console. It is among other available on the Trinity Rescue Kit. For instance, to fix the boot record of a windows partition at /dev/sda2 issue the commands:
relocntfs /dev/sda2
relocntfs -w /dev/sda2 # To actually write the new boot record
- stat
- display file or file system status (like file access time, creation time, modification time...)
- tree
- List the contents of directories in a tree-like format.
Network
autossh
autossh - Automatically restart SSH sessions and tunnels.
autossh by default monitors the ssh connection through a dedicated port to see whether the current ssh connection must be restarted. However the simpler is to use the ssh config option ServerAliveCountMax and ServerAliveInterval so that ssh exits automatically when the connection is down. In that case, autossh will then restart ssh automatically without need of additional monitoring.
Add to your ~/.ssh/config:
Host * ServerAliveCountMax 3 # default value actually ServerAliveInterval 15 # ssh will exit after 3 x 15s = 45s
Example of uses:
$ autossh -M 0 -f -D 1080 noekeon # -M 0 tells autossh to only restart ssh upon ssh's exit
dig
dig stands for domain information groper. It replaces the deprecated nslookup. It comes with package dnsutils:
sudo apt-get install dnsutils
ip
ip show / manipulate routing, devices, policy routing and tunnels:
IPETH0=$(ip addr show eth0 | perl -lne 'print for /inet[^0-9]*([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)/') # Get local ip address
multitee
multitee sends multiple inputs to multiple outputs.
Check this page.
Original is here, or can also be found here on Debian.
Here a patch to build it on Cygwin. The patch ports calls to BSD signal.h API (struct sigvec, sigvec(), sigmask(), sigblock()...) to the POSIX API (struct sigaction, sigaction(), sigprocmask(), sigfillset()...):
$ patch -Np1 <../multitee-3.0-cygwin.patch
$ make
$ cp multitee.1 /usr/local/man/man1
$ cp multitee.exe /usr/local/bin
Example of use:
$ multitee 0-1,4,5 4> foo 5> bar # same as tee foo bar with better blocking behavior
$ multitee 0:1 3:1 4:1,2 6:7 # various merge and copy
$ tcpclient server smtp multitee 0:7 6:1e0 # e0 tell multitee to quit as soon connection close
$ multitee 0:3 3:1 # same as 'socat - FD:3'
netcat
TCP-IP swiss army knife (equivalent of the telnet program. Check wikipedia:netcat. Also known as command nc).
See also Using Netcat (Linux Tips).
netstat
Print network connections, routing tables, interface statistics, masqurade connections, and multicast memberships
netstat -utpn #Active ports, tcp, socket program PID, numeric
netstat -lutpn #Listen ports, tcp, socket program PID, numeric
netstat -autpn #All (active and listen), tcp, socket program PID, numeric
netstat -rn #Kernel route table, numeric
When listing sockets (default output), you'll get an output like:
% netstat -at Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 *:time *:* LISTEN tcp 0 0 localhost:mysql *:* LISTEN tcp 0 0 andLinux.local:43449 windows-host:x11 ESTABLISHED |
% netstat -atn Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:37 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN tcp 0 0 192.168.11.150:43449 192.168.11.1:6000 ESTABLISHED |
- Local Address
- * or 0.0.0.0 means that the process accepts connection from any interface.
- 127.0.0.1 means it only accepts connection on localhost loopback (and so only connection that originates from local PC as well).
- Any other IP address means that the process listen on the given port at the given IP address
nmap
nmap is the network exploration tool and security / port scanner.
nmap localhost # Scan and print all open ports on 'localhost'
nmap -sP hostname # Simply test whether 'hostname' is accessible
One can use nmap in a bash script to test whether a given host is accessible, but it turns out that ping is faster:
# Using nmap
nmap -sP $1 2>/dev/null | grep -q "1 host up" # Too slow, does not always detect hosts
# ... or ping (faster)
if [ "$HOST_MACH" = "cygwin" ]; then
ping $1 1 1 2>/dev/null | grep -q "1 packets received"
else
ping -c 1 $1 2>/dev/null | grep -q "1 received"
fi
nm-tool
nm-tool is an utility to report NetworkManager state and devices.
nm-tool # Print IP addresses, DNS servers, etc.
/etc/init.d/nscd
This is not really a command, but the init service Name Service Cache Daemon.
sudo /etc/init.d/nscd restart # To restart daemon and flush DNS cache
rpcinfo
rpcinfo reports RPC information
rpcinfo -p # Probe the portmapper on host, and print a list of all registered RPC programs
socat
Command-line utility that establishes two bidirectional byte streams and transfers data between them ([8]). socat is the more powerful version of netcat. Check the homepage. And also this page on Yobi for examples on how to use socat to bypass a proxy.
# All these commands listen on localhost:53805, and forward to server:53806
socat TCP4-LISTEN:53805 TCP:server:53806 # Listen, and close as soon socket closes
socat TCP4-LISTEN:53805,reuseaddr,fork TCP:server:53806 # Idem, but fork so that listener continues after socket closes
socat -v ... # Send txfed data to stderr, with some formatting (text format)
socat -x ... # ... idem in hex format (can be combined with -v)
socat -ly # Writes messages to syslog instead of stderr; severity as defined with option -d
socat can easily tunnel connections:
socat -ly 'TCP4-LISTEN:143,reuseaddr,fork' PROXY:ton.imap.server:143|TCP:134.27.168.36:8080
socat can also be used as SSH ProxyCommand:
ProxyCommand socat -ly - 'PROXY:%h:%p|TCP:proxy.server:8080'
# Using socat v2.0.0 beta
ProxyCommand socat -ly - 'PROXY:%h:%p,proxyauth=user:pass|SSL,verify=0|PROXY:my.server:443,proxyauth=user:pass|TCP:big.brother.proxy:8080'
socat can be easily used as a replacement of telnet:
socat tcp:<host>:<port> - # <port> can be a port number or service name (telnet,imap...)
QUESTION TO MR DOEGOX:
- WHY -s in SED?
- Where does it get info on % for socat?
$ socat echo -
$ socat echo STDIO
$ socat echo STDOUT%STDIN
$ socat echo STDIO%STDIO
$ socat echo -%-
$ socat echo FD:1%FD:0
$ socat echo 1%0
/usr/local/bin/socat TCP:127.0.0.1:1234 -%EXEC:"awk 'BEGIN{print \"BANNER\";fflush()}/BANNER/{next}//{print;fflush()}'"
Testing socat:
- /usr/local/bin/socat - TCP-LISTEN:1234
- /usr/local/bin/socat TCP:127.0.0.1:1234 EXEC:"/bin/sed -s 's/foo/bar/g'"%EXEC:"/bin/sed -u 's/toto/tata/g'"
First open a port for listening and connect it to STDIO
In another shell, connect and do the black magic...
reference:
- http://eros.ph0k.eu/~nizox/twin/V2/modules/socat-1.6.0.0/EXAMPLES
- http://www.rschulz.eu/2008/09/ssh-proxycommand-without-netcat.html
$ exec 3<>/dev/tcp/zeus.yobi.be/22;(cat <&3 & );cat >&3
SSH-2.0-OpenSSH_5.1p1 Debian-3
- http://thesmithfam.org/blog/2006/05/23/bash-socket-programming-with-devtcp-2/
- http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=146464
- http://www.mtu.net/~engstrom/ssh-proxy.php
ATTEMPT 3:
Window 1: socat -v -%'EXEC:/home/beq06659/.ssh/mys2.sh' TCP-LISTEN:1234
Window 2: /usr/local/bin/socat -ly -v -%"EXEC:sed -unf /home/beq06659/.ssh/sedy" "PROXY:ftp.noekeon.org:22|TCP:localhost:1234"
File mys2.sh:
#! /bin/bash
echo -e "HTTP/1.1 200 Connection established\r\n\r\n"
cat
File sedy:
1 s/^/SSH-2.0-OpenSSH_5.1\n/p; /SSH-2.0-OpenSSH_5.1/d; /SSH-2.0-OpenSSH_5.1/! p
LESSONS LEARNED:
- READ THE CHANGES DOC, especially for BETA SW
- SOCKET = bidir --> a filter = 4 descriptors
- PIPE = unidir --> a filter = 2 descriptor (stdin/stdout)
Internet
httrack
httrack is a offline browser; it copies websites to a local directory.
# Example: checking out progit boot
# -d stay on same principal domain
# -w mirror web sites
# -O /var/www path for mirror/logfiles+cache
# -I0 don't make an index
# -%v display on screen filesnames downloaded
# -P 127.0.0.1:8118 Proxy settings
# '+progit.org/figures/*' +filter
# '-progit.org/book/*/*' -filter
httrack http://progit.org/book/ -d -w -O /var/www -I0 -%v '+progit.org/figures/*' '-progit.org/book/*/*' [-P 127.0.0.1:8118]
sed -ri \"s_'http://progit.org/book'_'index.html'_\" /var/www/progit.org/book/*.html
# Example 2
# -W mirror web sites, semi-automatic (ask question)
httrack http://www.cplusplus.com/ -W -O /var/www -%v '-www.cplusplus.com/forum/*' '-www.cplusplus.com/src/*' '-www.cplusplus.com/member/*' [-P 127.0.0.1:8118]
# Example 3 - Use different path for cache/log files
# -O /var/www,/tmp/httrack path for mirror,path cache and logfiles
httrack http://www.argospress.com/Resources/systems-engineering/index.html -d -O /var/www,/tmp/httrack -I0 -%v [-P 127.0.0.1:8118]
sed -ri '/wms.assoc-amazon.com/{N; d}' /var/www/www.argospress.com/Resources/systems-engineering/*.htm
# Example 4 - Limit depth
# -r2 set mirror depth to 2 (i.e. given address + 1st links)
httrack http://powerman.name/doc/asciidoc -r2 -w -I0 [-P 127.0.0.1:8118]
Serial
stty
stty changes and prints terminal line settings. It usually comes pre-installed.
stty -F /dev/ttyUSB0 -a # Print all settings in human-readable form
stty -F /dev/ttyUSB0 --save # Print all settings in stty-readable form
setserial
setserial get/set Linux serial port information.
See for instance [9] for usage.
Package Management
dpkg
Package manager for Debian / Ubuntu. See Package Management#Ubuntu / Debian
rpm
System
dmidecode
dmidecode is a tool to decode DMI (SMBIOS) tables:
sudo dmidecode
...
Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: Dell Inc.
Version: A19
Release Date: 12/21/2009
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 1728 kB
Characteristics:
ISA is supported
PCI is supported
PC Card (PCMCIA) is supported
PNP is supported
BIOS is upgradeable
BIOS shadowing is allowed
...
iwconfig
iwconfig configures or queries a wireless network interface:
iwconfig
# # iwconfig
# lo no wireless extensions.
#
# eth0 no wireless extensions.
#
# wlan0 IEEE 802.11abg ESSID:off/any
# Mode:Managed Access Point: Not-Associated Tx-Power=15 dBm
# Retry long limit:7 RTS thr:off Fragment thr:off
# Encryption key:off
# Power Management:off
iwconfig wlan0 # Limit to wlan0
lsmod / modinfo / modprobe
lsmod # Show the status of modules in the Linux Kernel
modinfo iwlagn # Show information about a Linux Kernel module
modprobe -r iwlagn # Remove module from the Linux Kernel
modprobe iwlagn 11n_disable=1 # Add module to the Linux Kernel, with specific parameters
loadkeys
loadkeys loads keyboard translation tables.
Use loadkeys to change the keyboard layout in console mode (i.e. Ctrl-Alt-F1 console).
sudo loadkeys be # Load Belgian layout
lspci
lspci # List all PCI devices
lspci -v # List all PCI devices (verbose)
newgrp
newgrp log in to a new group
newgrp - <group> # Start a new session with <group> as primary group. - means to reinitialize environment.
umask 002 # Make sure all files / directories we create have group access
To run newgrp in a script, use the following construction [10]:
newgrp groupb << END
touch "myscript_output.txt"
END
Or better yet use the sg command instead.
pidinfo
Find the process ID of a running program
pidof apache2 # Find all pid of apache2
pgrep
Prints PID of the process matching given pattern. Never match pgrep itself!
ps -fp $(pgrep autossh) # Much better that "ps -aef | grep autossh", which match grep process itself
pkill
Send a signal (default TERM) to process matching given pattern. Never match pkill itself!
pkill -0 autossh # Test if process "autossh" is running
ps
ps -ef # All processes - full format
ps afl # Proc w ttys, all users - long + forest
ps flx # Proc w/o ttys - long + forest
ps aflx # All processes, all users - long + forest
ps au # Proc w ttys, all users - user output
ps ux # Proc w/o ttys - user output
ps aux # All processes, all users - user output
ps axs # All process, with signals
pv
pipe viewer — terminal-based tool for monitoring the progress of data through a pipeline
pv -cN source < linux-2.2.20.tar.bz2 | bzcat | pv -CN bzcat | gzip -9 | pv -cN gzip > linux-2.2.20.tar.gz
source: 4.48MB 0:02:01 [9.84kB/s] [========> ] 29% ETA 0:04:44
bzcat: 26MB 0:02:01 [ 192kB/s] [ <=> ]
gzip: 5.58MB 0:02:01 [ 64kB/s] [ <=> ]
sg
sg executes command as different group ID.
umask 002 # Ensure that all files written will be group accessible
sg - <group> -c "mkdir foo" # Run commands as group <group>
The following construction restart the current script under a different group (from [11]):
if ! [ "${SBREADY:=false}" = true -o $(id -ng) = ${NEEDGRP:=wheel} ] ; then
export SBREADY=true
exec sg $NEEDGRP "$0" "$@"
fi
sudo / gksudo
See also Ubuntu's page at [linux reference script security sudo ubuntu].
ls | sudo tee /root/somefile # Instead of redirection sudo ls > /root/somefile
ls | sudo tee -a /root/somefile # ... same but append instead
sudo sh -c "ls > /root/somefile" # Yet another way to do redirection with sudo
sudo !! # Repeat last command with root privilege
sudo -i # Get a root *login* shell
sudo su - # ... another solution
sudo -s # Root shell but keeping current environment
sudo su # ... another solution
sudo -k # Reset sudo password timeout
gksudo is the graphical equivalent of sudo:
gksudo gedit /etc/fstab
showkey
showkey examine the codes sent by the keyboard (linux kernel only - for X keyboard events, see xev!).
showkey -k # Dump key codes (default)
showkey -s # Dump scan codes
timeout / timelimit
timeout / timelimit are small binary to run a command with a given time limit. More exactly they send a given signal to the child process after some specified time. See also Perl - kill on ALARM signal).
timeout -15 30 mplayer dvd:// # Show 30s of DVD movie, then kill mplayer with a TERM signal
- free
- Display the amount of free and used memory (free -m to get info in MB)
free
free -m # Show in megabyte
- /proc/meminfo
- All memory usage information
cat /proc/meminfo
- vmstat
- Summary of memory usage
vmstat
Miscellaneous
- htop
- an improved top command
- lsb_release
- lsb_release -a prints version information for the Linux release you're running.
- mkfifo
- make FIFOs (named pipes)
- pv, pipeview, pipebench
- monitor the progress of data through a pipe
- strace
- trace system calls and signals
- tee
- read from standard input and write to standard output and files
- uname
- uname -a prints all system information, include machine name, kernel name & version.
- watch
- Execute a program periodically, showing output full screen
User / Group Administration
groupadd / addgroup
Use groupadd, addgroup to create a new group. groupadd is the low-level utility. addgroup is the Debian more friendly version.
sudo groupadd groupname # Create a new group groupname
sudo groupadd -g 123 groupname # ... and specify gid of the new group
sudo addgroup groupname
sudo addgroup --gid 123 groupname
useradd / adduser
Use useradd to create new user. useradd is the low-level utility. adduser is the Debian more friendly version.
sudo useradd username # Create a new user username
sudo useradd -s /bin/bash username # ... specify the deafult shell in use for the user
sudo useradd -s /bin/bash -m username # ... and create user home directory (from /etc/skel)
sudo useradd -u 1234 username # Create a new user username with uid 1234
sudo useradd -g GRP1 -G GRP2,GRP3 username # Specify primary and supplementary groups
sudo adduser username # Create a new user username
sudo adduser --system username # Create a new system user (in the system uid range)
sudo adduser --shell /bin/bash username # Use /bin/bash as default login shell
userdel / groupdel / deluser
Use userdel, groupdel, deluser to remove an existing user /group. deluser is the Debian more friendly version
sudo userdel username # Remove existing user username
sudo groupdel groupname # Remove existing group groupname
sudo deluser username # Remove existing user username
sudo deluser --group groupname # Remove existing group groupname
usermod
Use usermod to modify an existing user
sudo usermod -g GRP1 username # Modify the primary group of user username
sudo usermod -g GRP1 -G GRP2,GRP3 username # ... and also the supplementary groups
gpasswd
Use gpasswd to administer /etc/group and /etc/gshadow
sudo gpasswd -a $USER fuse # Add self to group 'fuse'
X
disper
TO install:
sudo add-apt-repository ppa:wvengen/ppa
sudo apt-get update
sudo apt-get install
Some examples of use:
disper -s # Only enable primary screen (laptop screen)
disper -S # Only enable secondary screen (external monitor)
disper -d auto -e # Auto-detect + enable extended mode
disper -d auto -c # clone mode
xbindkeys / xbindkeys-config
- xbindkeys is a program that allows you to launch shell commands with your keyboard or your mouse under the X Window System. It links commands to keys or mouse buttons, using a configuration file. It's independent of the window manager and can capture all keyboard keys (ex: Power, Wake...).
- xbindkeys-config is an easy to use gtk program for configuring Xbindkeys (see [12].
xclip
- xclip is a command line interface to the X11 clipboard. It can also be used for copying files, as an alternative to sftp/scp, thus avoiding password prompts when X11 forwarding has already been setup. Check this guide.
xdpyinfo
Display information utility for X
xdpyinfo | grep dimensions # Return dimension of the desktop
XSel / XSelection
XSel is a command-line program for getting and setting the contents of the X selection and context-menu clipboard. Normally the former is only accessible by manually highlighting information and pasting it with the middle mouse button, and the latter is only accessible by using context-menu cut/copy/paste command.
xselection is an equivalent package on OpenSUSE.
On Cygwin, one has to use getclip, putclip or device /dev/clipboard.
xsel -p # To get content of PRIMARY buffer (X selection)
xselection PRIMARY # (equivalent command for xselection)
xsel -b # To get content of CLIPBOARD (context-menu clipboard)
xselection CLIPBOARD # (equivalent command for xselection)
echo sometext | xsel -p # To set PRIMARY buffer
echo sometext | xselection PRIMARY - # (equivalent command for xselection)
xselection PRIMARY sometext # (equivalent command for xselection)
echo sometext | xsel -b # To set CLIPBOARD buffer
echo sometext | xselection CLIPBOARD - # (equivalent command for xselection)
xselection CLIPBOARD sometext # (equivalent command for xselection)
xvkbd
xvkbd is a virtual (graphical) keyboard program for X Window System which provides facility to enter characters onto other clients (softwares) by clicking on a keyboard displayed on the screen. It can also be used to send keyboard events through command-line:
xvkbd -xsendevent -text text-string
xwininfo
xwininfo is the window information utility for X. Could be used to find the geometry of a window. See here
xwininfo # Return information on the clicked windows
xwininfo -root # ... or on the root window
xwininfo -root |awk '/Width/{print $2} # Get Width of the current desktop
Text Manipulation
column
Format input into multiple columns
cat | column -l
123 456
98765 98765
^D
# 123 456
# 98765 98765
cut
TBC
paste
TBC
fold
fold -w 40 FILE # Wrap FILE to 40-char width
expand
expand -t 4 FILE # Convert tabs to spaces (4)
expand -t 4,10 FILE # Set explicit tab positions
fmt
TBC
nl
nl numbers lines of files
nl -ba FILE # Number *all* lines of FILE
rl, shuf
rl is a command-line tool that reads lines from an input file or stdin, randomizes the lines and outputs a specified number of lines (requires package randomize-lines).
shuf generates random permutations.
They are alternative to sort -R (note that the latter does not work with duplicate lines).
sed
Moved to page dedicated to Sed.
sort
sort is the well-know text sort utility.
sort -R shuffle.txt # shuffle the lines (use random hash) - fail with duplicate lines
nl -ba file | sort -R | sed 's/^.*[0-9]\t//' # ... idem, but workaround for dup lines
sort by default takes into account the current locale. To get the traditional sort order that uses native byte values:
LC_ALL=C sort myfile.txt # traditional sort, indep of current locale
Use -o file to sort in-place, or use the sponge utility:
sort file -o file # sort in-place
sort file | sponge file # soak sorted file and write it back
Development
addr2line
addr2Line converts addresses into file names and line numbers.
ar
ar creates, modifies, and extracts from archives. It can be used to generate static libraries (.a files) for use with gcc.
diff
Use diff to compare 2 files/directories together and/or to generate patch files. Use colordiff (package colordiff) to have colored output.!
diff old new # View difference (default ed script format)
diff -u old new >new.patch # Generate a patch file (universal format)
diff -y old new # View difference side-by-side (as with sdiff)
#Color with DIFF:
colordiff -u old new # colordiff can also colorize a diff output package colordiff
diff old new | colordiff # ... idem
#Color with WDIFF:
wdiff -n old new | colordiff # But need option '-n'
#Piping to LESS with COLORS
diff -u old new | colordiff | less -R # To keep color with pager
#DIFF / WDIFF combo
diff --strip-trailing-cr -bu old new | wdiff -d | colordiff | less -R # Handy when wdiff miss some options...
Patch can be applied with the command patch (see below). Be careful regarding file names / directory structures when generating patch files! Sometimes you'll need to edit the generated patch files to fix the directory structure.
patch
Use patch to apply a patch previously generated with the command diff. patch use is pretty much straightforward, one mainly has to pay attention to directory structure / blanks.
Complete example using diff:
$ cp project project-patched
$ cd project-patched
$ vi somefile # We start modifying the copy
$ cd ..
$ diff -u project project-patched >project.patch # We generate the patch file (universal format)
$ cd project # We cd into project directory to patch
$ patch -lNp1<../project.patch # We apply the patch in place
- -l — ignore white spaces (very handy when copying patch from internet page for instance).
- -N — ignore patch that seems reversed or already applied (same as --forward).
- -p1 — tells patch to ignore the first (i.e. one level of) directory in the file name given in the patch header.
The value to use for option p depends actually on the patch header:
- First example - same root directory for both to and from file
- second example - different root directory between to and from file
--- outguess-0.2/jpg.c 2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2/jpg.c 2009-08-25 16:06:05.242378300 +0200
$ patch -lNp0 <project.patch # First example
--- outguess-0.2/jpg.c 2001-02-13 01:29:07.000000000 +0100
+++ outguess-0.2-patched/jpg.c 2009-08-25 16:06:05.242378300 +0200
$ cd project_dir; patch -lNp1 <../project.patch # Second example
vbindiff
vbindiff — hexadecimal file display and comparison
vbindiff file1.bin file2.bin # Compare and view 2 binary files
Miscellaneous
cksum / jacksum / md5sum / sha1sum / sum
cksum file # Generate a CRC similar to CRC32, but in decimal, and with length appended.
cksum -o3 file # Generate a CRC32 checksum (BSD only)
sum file # Generate a checksum
md5sum file # Hash file using MD5
sha1sum file # Hash file using SHA1
jacksum -a crc32 file # Hash using CRC32 (dec)
jacksum -a crc32 -E hexup file # Hash using CRC32 (hex)
dog
dog is better than cat. It writes the contents of each give file, URL, or the standard input to standard output.
dtach
dtach simple program that emulates the detach feature of screen. A session in dtach is a single instance in which a program is running under the control of dtach. The program is disassociated from the original terminal, and is thus protected from your original terminal being disconnected for some reason.
echo
echo -e "Some text\n...on 2 lines..." # Enable interpretation of backslash escapes (must be quoted!)
expect
Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.
iconv
locale encoding conversions
konwert
fancy encoding conversions
mc (Midnight Commander)
mc or Midnight Commander is a powerful file manager working in a shell terminal.
See dedicated page on this wiki.
mimencode
binary file conversion for the mail.
mrxvt
A multi-tab version of rxvt (terminal-emulator).
recode
recode /cl../cr <dos.txt >mac.txt
recode /cr.. <mac.txt >unix.txt
recode ../cl <unix.txt >dos.txt
recode charset1/surface1..charset2/surface2 <input.txt >output.txt
recode /QP.. <qp.txt >normal.txt # To convert quoted-printable text
| charset | surface | ||
|---|---|---|---|
| us | ASCII (7 bits) | /cr | Carriage return as end of line (Mac text) |
| l1 | ISO Latin-1 (ISO-8859-1, Western Europe, 8 bits) | /cl | Carriage return line feed as end of line (DOS text) |
| EUCJP | EUC-JP for Japanese (Unix) | / | Line feed as end of line (Unix text) |
| SJIS | Shift-JIS for Japanese (Microsoft) | /d1 | Human readable bytewise decimal dump |
| ISO2022JP | Mail encoding for Japanese (7 bits) | /x1 | Human readable bytewise hexidecimal dump |
| u2 | UCS-2 (Universal Character Set, 2 bytes) | /64 | Base64 encoded text |
| u8 | UTF-8 (Universal Transformation Format, 8 bits) | /QP | Quoted-Printable encoded text |
reformime
MIME E-mail reformatting tool
screen
Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.
See also tmux for a better alternatives that supports splitting in panes...
Launch with
screen # Start a new screen session
screen -r # Resumes a detacked screen session
screen -d -r # Reattach a session and if necessary detach it first
screen -d -R # Reattach a session and if necessary detach or even create it first
screen -D -r # Reattach a session. If necessary detach and logout remotely first
screen -D -R # Attach here and now, ie:
# - If no session is running, create it
# - Otherwise reattach the session. If necessary detach/logout remotely first
For convenience one may add the following alias to .bashrc:
alias screen=screen -D -R
Useful shortcuts:
| key | description |
|---|---|
| C-a c | Open a new instance |
| C-a Space | Switch to next instance |
| C-a d | Detach from screen. Reattach later with screen -r
|
| C-a D D | Power detach, i.e. detach and logout |
byobu is a wrapper around screen, which makes it prettier and more powerful (see this also).
sponge
sponge is part of package moreutils. It can be used to easily edit file in-place:
sed -r '...' FILE | grep ... | sponge FILE # Sponge soaks its full input before creating output file
tmux
tmux is actually a mix of screen and Gnome-Terminator. It is a terminal multiplexer: it enables a number of terminals to be created, accessed, and controlled from a single screen. Terminals can be organised in multiple panes in the same window.
Launch with
tmux # Start tmux (assume command 'new')
tmux attach # Start tmux and reattach to previous session
tmux complains if it is launched as an ssh command (error not a terminal), but this is fixed with option -t:
ssh -t server tmux attach # Launch tmux directly, require forcing terminal emulation
Useful shortcuts:
| key | description |
|---|---|
| C-b " C-b % |
Split horizontally Split vertically |
| C-b q |
Display pane id briefly |
| C-b d |
Detach from tmux. Reattach later with tmux attach
|
Attach automatically
One can configure tmux to attach automatically to an existing session, or create one if none exists:
- Add to ~/.tmux.conf
- Add to ~/.bashrc
new-session
# tmux - always attach (or pass given command and parameters)
# This solution requires to have the line "new-session" in ~/.tmux.conf
function tmux() {
T=/usr/bin/tmux
if [ $# -eq 0 ]; then $T attach; else $T "$@"; fi
}
Another solution (but not atomic) that does not require editing .tmux.conf is to use tmux attach || tmux new. This solutions also allows to use named sessions (see [13]).
Zoom on a pane
version 1.8 adds the zoom feature like terminator (see flag -Z to command resize-pane). Here a work-around for previous versions that binds the feature to C-b z (from [14]):
- Edit ~/.tmux.conf
- Edit ~/.tmux/zoom
unbind z
bind z run ". ~/.tmux/zoom"
#!/bin/bash -f
currentwindow=`tmux list-window | tr '\t' ' ' | sed -n -e '/(active)/s/^[^:]*: *\([^ ]*\) .*/\1/gp'`;
currentpane=`tmux list-panes | sed -n -e '/(active)/s/^\([^:]*\):.*/\1/gp'`;
panecount=`tmux list-panes | wc | sed -e 's/^ *//g' -e 's/ .*$//g'`;
inzoom=`echo $currentwindow | sed -n -e '/^zoom/p'`;
if [ $panecount -ne 1 ]; then
inzoom="";
fi
if [ $inzoom ]; then
lastpane=`echo $currentwindow | rev | cut -f 1 -d '@' | rev`;
lastwindow=`echo $currentwindow | cut -f 2- -d '@' | rev | cut -f 2- -d '@' | rev`;
tmux select-window -t $lastwindow;
tmux select-pane -t $lastpane;
tmux swap-pane -s $currentwindow;
tmux kill-window -t $currentwindow;
else
newwindowname=zoom@$currentwindow@$currentpane;
tmux new-window -d -n $newwindowname;
tmux swap-pane -s $newwindowname;
tmux select-window -t $newwindowname;
fi
Scroll with mouse wheel
Add to ~/.tmux.conf:
#set-window-option -g mode-mouse on # Does not work when ssh from cygwin
set -g terminal-overrides 'xterm*:smcup@:rmcup@'
uuencode, uudecode
Binary file conversion for Unix.
wc
Counts lines, words and character in a file
xargs
For more examples, see combinations with the command find
#By default, xargs cuts at WHITE SPACE and NEWLINE. Use -d '\n' to only cut at NEWLINES
for i in $(locate .pdf); do basename $i; done # WRONG
locate .pdf | xargs -d '\n' -n 1 basemane # Correct
Hex Tools
hd / hexdump
ASCII, decimal, hexadecimal, octal dump.
hexdump -e '"%2x"' <myfile> # Convert myfile into a long hexadecimal string - ! See DOUBLE-QUOTED parameter
hexdump -C <myfile> # Canonical hex + ascii display (16 bytes)
hd <myfile> # (idem)
See also od and xxd. There are also hex editors, like hexer and hexedit (adapts to width of the terminal!).
od
od - dump files in octal and other formats
od -xc file # Output file in ascii and hex output (2-byte, ! little-endian)
od -t x1 -a --width=32 testvar.sh # Output 32-byte at once
See also hd / hexdump and xxd.
xxd
Make a hexdump or do the reverse. This tool is delivered along with vim.
echo 202122 | xxd -r -p # Convert hexdump to a binary string
echo -n ' !"' | xxd -p # Convert binary string to hexdump - DON'T FORGET -n
echo -n ' !"' | xxd -u -p # Convert binary string to hexdump - DON'T FORGET -n (uppercase)
See also hd / hexdump and od.