Linux Admin: Difference between revisions
(→Backup) |
|||
(128 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
==Documentation / Getting Help== |
== Documentation / Getting Help == |
||
;yelp |
;yelp |
||
:Default Gnome help system. Contains basic documentation, manpages, and guides (which can even install applications if clicked on) |
:Default Gnome help system. Contains basic documentation, manpages, and guides (which can even install applications if clicked on) |
||
Line 68: | Line 68: | ||
</source> |
</source> |
||
== File Systems == |
== File Systems & Backup == |
||
See page [[Linux Disk Management]]. |
See page [[Linux Disk Management]]. |
||
== sysfs and configfs == |
|||
Reference: |
|||
* [http://www.linux.org/threads/sysfs-and-configfs.4956/ Sysfs and Configfs] |
|||
The virtual filesystem '''sysfs''' contains device and driver information as well as various kernel data. sysfs |
|||
provides an interface between kernelspace and userspace. |
|||
;{{file|/sys/block/}} : This directory contains shortcuts to the sysfs files of the selected block device. |
|||
;{{file|/sys/bus/}} : This directory contains the sysfs files and data for the different buses on the system. |
|||
;{{file|/sys/class/}} : This directory contains folders named by device type like "printers", "mem", "leds", "input", etc. The subdirectories then contain shortcuts to the sysfs files pertaining to the selected device. This is useful for when a user is looking for a particular device. They can then go through this route. |
|||
;{{file|/sys/dev/}} : Inside, there are two folders - "block" and "char" which direct users to the block and character devices respectively. |
|||
;{{file|/sys/devices/}} : Most of the symbolic/soft links (shortcuts) in the sysfs system link to devices and files here. |
|||
;{{file|/sys/firmware/}} : Files containing information and settings for the system's firmware resides here. |
|||
;{{file|/sys/fs/}} : Information and settings pertaining to each individual mounted filesystem is placed here by filesystem type. This means there is a folder titled "ext4" which holds the folders for each device/partition with ext4. |
|||
;{{file|/sys/hypervisor/}} : Data pertaining to the hypervisor is stored here. |
|||
;{{file|/sys/kernel/}} : Settings, information, and security policies reside in this folder. |
|||
;{{file|/sys/module/}} : All of the loaded modules can be seen here. Each folder contains information and settings for that particular module. |
|||
;{{file|/sys/power/}} : This directory contains files with information on the power state, the number of times the system hibernated/slept, etc. |
|||
'''configfs''' is also a virtual filesystem on memory. configfs is used to change, make, and delete kernel objects and data. |
|||
configfs is a kernel manager in the form of a filesystem. |
|||
To mount ''configfs'': |
|||
<source lang=bash> |
|||
sudo mount -t configfs none /sys/kernel/config |
|||
</source> |
|||
Settings in configfs are usually set as (with root privilege): |
|||
<source lang=bash> |
|||
echo "STRING" > /config/FILE/PATH |
|||
</source> |
|||
== System == |
== System == |
||
* [http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html 20 Linux System Monitoring Tools Every SysAdmin Should Know] |
|||
* [https://www.tecmint.com/command-line-tools-to-monitor-linux-performance/ 20 Command Line Tools to Monitor Linux Performance] |
|||
: <code>top</code>, <code>vmstat</code>, <code>lsof</code>, <code>tcpdump</code>, <code>netstat</code>, <code>htop</code>, <code>iotop</code>, <code>iostat</code>, <code>iptraf</code>, <code>psacct</code> or <code>acct</code>, Monit, <code>nethogs</code>, <code>iftop</code>, Monitorix, Arpwatch, Suricata, VnStat PHP, NMon, Collectl. |
|||
Some more tools: |
|||
* [http://www.cyberciti.biz/tips/linux-scanning-network-for-open-ports.html <code>nmap</code>] |
|||
* [http://members.dslextreme.com/users/billw/gkrellm/gkrellm.html <code>GKrellM</code>] |
|||
* [http://www.cyberciti.biz/tips/keeping-a-log-of-daily-network-traffic-for-adsl-or-dedicated-remote-linux-box.html <code>vnstat</code>] |
|||
* [http://www.cyberciti.biz/tips/finding-out-a-bad-or-simply-overloaded-network-link-with-linuxunix-oses.html <code>mtr</code>] |
|||
* [http://dag.wiee.rs/home-made/dstat/ <code>dstat</code>] (combines vmstat, iostat, ifstat, netstat information and more) |
|||
=== System monitoring === |
|||
* '''(CLI)''' <code>ps</code> — displays the processes |
|||
* '''(CLI)''' <code>free</code> — Display amount of free and used memory in the system |
|||
* '''(CLI)''' <code>vmstat</code> — system activity, hardware and system information |
|||
* '''(CLI)''' <code>strace</code> — trace system calls and signals |
|||
* '''(CLI)''' <code>ltrace</code> — trace calls to dynamic libraries |
|||
* {{file|/proc}} file system — various kernel statistics |
|||
<source lang=bash> |
|||
cat /proc/stat |
|||
cat /proc/$PID/stat |
|||
cat /proc/cpuinfo |
|||
cat /proc/meminfo |
|||
cat /proc/zoneinfo |
|||
cat /proc/mounts |
|||
</source> |
|||
* '''(CLI)''' <code>mpstat</code> — multiprocessor usage |
|||
: Part of {{deb|sysstat}} package. |
|||
* '''(CLI)''' <code>pmap</code> — report memory map of a process |
|||
* '''(CLI)''' [https://www.selenic.com/smem/ <code>smem</code>] a more clever tool to monitor memory usage |
|||
: Can report RSS, but also PSS (proportional set size), a more meaninful representation of memory used [http://stackoverflow.com/questions/4802481/how-to-see-top-processes-by-actual-memory-usage] |
|||
: {{green|Nice!}} |
|||
* '''(TUI)''' <code>top</code> — process activity command |
|||
* '''(TUI)''' <code>atop</code> — AT Computing's System & Process Monitor |
|||
: '''{{red|BEST!}}''' An even better ''top'' program, with disk io stats, busy rate... Very easy to see what's going on. Use {{kb|h}} for help, {{kb|t}} to trigger update, {{kb|1}} to switch to value/sec. |
|||
* '''(TUI)''' [http://hisham.hm/htop/ <code>htop</code>] — an interactive process viewer for Linux ([http://htop.sourceforge.net/ SourceForge]) |
|||
: {{green|Nice!}} An improved ''top'' program. |
|||
* '''(CLI)''' <code>sar</code> — collect and report system activity |
|||
: Part of {{deb|sysstat}} package. |
|||
* '''sysstat''' suite, containing <code>sar</code>, <code>iostat</code>, <code>tapestat</code>, <code>mpstat</code>, <code>pidstat</code>, <code>sadf</code>, <code>cifsiostat</code>. |
|||
: '''{{green|Nice!}}''' With data collection... |
|||
* '''(GUI)''' KDE System Guard - Real-time Systems Reporting and Graphing |
|||
* '''(GUI)''' Gnome System Monitor - Real-time Systems Reporting and Graphing |
|||
* '''(GUI)''' [https://github.com/brndnmtthws/conky <code>Conky</code>] — Lightweight system monitoring for X |
|||
* '''(Web-UI)''' Nagios Server And Network Monitoring |
|||
* '''(Web-UI)''' Cacti - Web-based Monitoring Tool |
|||
* '''(Web-UI)''' [http://munin-monitoring.org/ <code>munin</code>] |
|||
* '''BPF-based''': [https://0x.tools/ 0x.tools], [https://github.com/elastic/otel-profiling-agent otel-profiling-agent] |
|||
:* More info on BPF: [https://brendangregg.com/ebpf.html Linux Extended BPF (eBPF) Tracing Tools]. See also [https://news.ycombinator.com/item?id=40869877 HN comments] |
|||
=== Disk / file monitoring === |
|||
* '''(CLI)''' <code>iostat</code> — average cpu load, disk activity |
|||
: Part of {{deb|sysstat}} package. |
|||
* [http://www.cyberciti.biz/tips/tag/lsof-command <code>lsof</code>] — list open files |
|||
<source lang=bash> |
|||
lsof -f | grep bash |
|||
lsof -p $$ # view all files opened by current shell |
|||
</source> |
|||
=== Network monitoring === |
|||
* '''(CLI)''' <code>netstat</code> — Print network connections, routing tables, interface statistics, masquerade connections, and multicast membershipsand |
|||
* '''(CLI)''' <code>ss</code> — another utility to investigate sockets |
|||
<source lang="bash"> |
|||
ss -tupan # Roughly equivalent to netstat -lpn |
|||
</source> |
|||
* '''(CLI)''' <code>iptraf</code> — real-time network statistics |
|||
* '''(CLI)''' <code>tcpdump</code> — detailed network traffic analysis |
|||
* [http://gael.roualland.free.fr/ifstat/ <code>ifstat</code>] — shows network traffic by interface in a vmstat/iostat-like manner |
|||
* '''(CLI)''' <code>nethogs</code> — Nethogs monitors traffic going to/from a machine, per process. |
|||
: {{green|Nice!}} |
|||
* '''(TUI)''' <code>iotop</code> — simple top-like I/O monitor |
|||
* '''(TUI)''' [https://github.com/tgraf/bmon <code>bmon</code>] — bandwidth monitor and rate estimator |
|||
: {{green|Nice!}} A kind of console mode darkstat. |
|||
* '''(TUI)''' [http://ex-parrot.com/~pdw/iftop/ <code>iftop</code>] — display bandwidth usage on an interface |
|||
:shows network traffic by service and host |
|||
* '''(GUI)''' [https://unix4lyfe.org/darkstat/ darkstat]— Captures network traffic, calculates statistics about usage, and serves reports over HTTP. |
|||
: Breaks down traffic by host, protocol, etc. Geared towards analysing traffic gathered over a longer period, rather than `live' viewing. |
|||
: {{green|Nice!}} |
|||
* '''(Web-UI)''' ntopng — High-speed web-based traffic analysis and flow collection tool |
|||
: See [[ntopng]]. |
|||
* '''(Web-UI)''' [http://www.cyberciti.biz/faq/debian-ubuntu-install-ntop-network-traffic-monitoring-software/ <code>ntop</code>] |
|||
=== Network tools === |
|||
* '''[http://ettercap.github.io/ettercap/ ettercap]''' is a network sniffer/interceptor/logger for ethernet |
|||
=== Other tools === |
|||
* <code>w</code> — find out who is logged on and what they are doing |
|||
* <code>uptime</code> — tell how long the system has been running |
|||
* <code>uptimed</code> and <code>uprecords</code> — daemon to record uptime records |
|||
<source lang="bash"> |
|||
uprecords -a -B |
|||
# # Uptime | System Boot up |
|||
# ----------------------------+--------------------------------------------------- |
|||
# -> 1 0 days, 00:35:30 | Linux 4.19.0-2-amd64 Wed Mar 27 09:12:36 2019 |
|||
# 2 1 day , 02:42:32 | Linux 4.19.0-2-amd64 Tue Mar 26 06:29:24 2019 |
|||
# 3 9 days, 09:16:49 | Linux 4.19.0-2-amd64 Sat Mar 16 21:12:05 2019 |
|||
</source> |
|||
=== Monitor a process activity === |
|||
* '''strace''' is the tool of choice to monitor a process activity. |
|||
* Detect library usage (to debug missing library for instance) |
|||
* Detect all IO activity |
|||
* And more... |
|||
<source lang=bash> |
|||
strace myprogram |
|||
strace -p $PID |
|||
strace -f myprogram # Follow child processes |
|||
</source> |
|||
* '''lsof''' (list open files) to view files opened by a given process, etc: |
|||
<source lang=bash> |
|||
lsof -f | grep rsync |
|||
lsof -p $$ |
|||
</source> |
|||
* '''ldd''' and '''readelf''' to list libraries |
|||
<source lang=bash> |
|||
ldd /bin/ls # List libraries loaded by given executable |
|||
readelf -a /bin/ls # List libraries, and much more |
|||
</source> |
|||
* Monitor access to log file. For instance with: |
|||
<source lang=bash> |
|||
ls -lS /var/log/*log | head |
|||
</source> |
|||
* See <code>apropos process</code> for even more tools. |
|||
<source lang=bash> |
|||
apropos process |
|||
# ... |
|||
# fuser (1) - identify processes using files or sockets |
|||
# ... |
|||
</source> |
|||
* [https://fabiensanglard.net/st/index.html st (space-time)] that monitors memory usage over time, and number of forks/threads. |
|||
=== Stop / continue a process === |
|||
<source lang="bash"> |
|||
pkill -TSTP process # STOP gracefully |
|||
pkill -STOP process # STOP forcefully |
|||
pkill -CONT process # Resume (CONT) a stopped process |
|||
</source> |
|||
=== IO |
=== IO usage tips === |
||
To view IO usage of a specific process, check '''<tt>/proc/self/io</tt>''', '''<tt>/proc/...pid.../io</tt>''': |
To view IO usage of a specific process, check '''<tt>/proc/self/io</tt>''', '''<tt>/proc/...pid.../io</tt>''': |
||
<source lang="bash"> |
<source lang="bash"> |
||
Line 86: | Line 291: | ||
# write_bytes: 0 |
# write_bytes: 0 |
||
# cancelled_write_bytes: 0 |
# cancelled_write_bytes: 0 |
||
</source> |
|||
=== Memory usage tips === |
|||
==== Free buffers and cache ==== |
|||
;To free buffers and cache [http://unix.stackexchange.com/questions/87908/how-do-you-empty-the-buffers-and-cache-on-a-linux-system]: |
|||
<source lang=bash> |
|||
free && sync && echo 3 > /proc/sys/vm/drop_caches && free |
|||
</source> |
|||
==== Get peak memory usage of a process ==== |
|||
From StackExchange [http://unix.stackexchange.com/questions/18841/measuring-ram-usage-of-a-program]: |
|||
* Use [https://github.com/gsauthof/cgmemtime cgmemtime], measures the high-water RSS+CACHE memory usage of a process and its descendant processes. |
|||
: Use kernel cgroup to measure memory usage of process '''and''' descendants: |
|||
<source lang="bash"> |
|||
sudo ./cgmemtime --setup -g myusergroup --perm 775 |
|||
# This created /sys/fs/cgroup/memory/cgmemtime |
|||
./cgmemtime ./testa x 10 20 30 |
|||
</source> |
|||
* Use [https://bitbucket.org/gsauthof/tstime tstime]. |
|||
<source lang=bash> |
|||
tstime date |
|||
</source> |
|||
* Use [https://gist.github.com/netj/526585/590293d6527c91e48fcb08edb8de9fd6c88a6d82 memusg], a script that does polling. |
|||
:However polling is a poor solution to measure peak statistics. |
|||
* Use standard <code>/usr/bin/time</code>: |
|||
<source lang=bash> |
|||
/usr/bin/time -v sort myfile |
|||
</source> |
|||
==== Manage Out-of-Memory Killer ==== |
|||
* https://www.oracle.com/technical-resources/articles/it-infrastructure/dev-oom-killer.html |
|||
When the system is getting out-of-memory (OOM), the kernel triggers the OOM killer. The OOM killer computes the <code>oom_score</code> of every process, and kill the process with highest score. This score can be adjusted through <code>oom_adj</code> (ranging from -16 to +15) or <code>oom_score_adj</code> (rangin from -1000 to +1000) |
|||
To see the <code>oom_score</code> of every process [https://askubuntu.com/questions/995248/oom-score-for-every-process]: |
|||
<source lang="bash"> |
|||
# Print directly from /proc |
|||
cat /proc/*/oom_score |
|||
# A nicer list with pid / oom_score / process name |
|||
while read -r pid comm; do |
|||
printf '%d\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$comm" |
|||
done < <(ps -e -o pid= -o comm=) |
|||
</source> |
|||
Likewise the <code>oom_adj</code> <code>oom_score_adj</code> are found in {{file|/proc}}: |
|||
<source lang="bash"> |
|||
cat /proc/*/oom_adj |
|||
cat /proc/*/oom_score_adj |
|||
</source> |
|||
To change the adjust value [https://askubuntu.com/questions/60672/how-do-i-use-oom-score-adj]: |
|||
<source lang="bash"> |
|||
# Change oom_score_adj of current process |
|||
echo 1000 > /proc/$$/oom_score_adj # OR |
|||
echo 1000 > /proc/self/oom_score_adj |
|||
# Change score on process startup |
|||
choom -n 1000 COMMAND [OPTION...] |
|||
</source> |
|||
Note that the <code>oom_score_adj</code> are '''inherited''' by forked processes. Also, setting negative values requires root access. |
|||
=== Send commands via D-BUS === |
|||
See [[D-Bus]]. |
|||
=== Power management === |
|||
See [[Power management]]. |
|||
=== BIOS update === |
|||
Using <code>fwupdmgr</code> (for system BIOS, Thunderbolt docking station...). |
|||
* Tested successfully on HP EliteBook G4 + Thunderbolt docking station. |
|||
<source lang="bash"> |
|||
# Refresh DB |
|||
sudo fwupdmgr refresh |
|||
# Get list of available update |
|||
sudo fwupdmgr get-upgrades |
|||
# Update all devices |
|||
sudo fwupdmgr update |
|||
</source> |
</source> |
||
== Hardware == |
== Hardware == |
||
=== |
=== Links === |
||
* [https://wiki.debian.org/HowToIdentifyADevice/PCI Debian - HowToIdentifyADevice/PCI] |
|||
: Speaks about <code>lspci</code> but also GUI tool '''hardinfo'''. |
|||
=== Info / troubleshooting commands === |
|||
Interesting commands: |
Interesting commands: |
||
* <tt>[[Linux Commands#lshw|lshw]]</tt>, list hardware |
* <tt>[[Linux Commands#lshw|lshw]]</tt>, list hardware |
||
Line 103: | Line 393: | ||
lspvi | grep -i vga # See video controller |
lspvi | grep -i vga # See video controller |
||
sudo lspci -vv -nn -s 0c: # Slot id 0c:... |
sudo lspci -vv -nn -s 0c: # Slot id 0c:... |
||
</source> |
|||
* <tt>[[Linux Commands#lsmem|lsmem]]</tt>, listing the ranges of available memory with their online status |
|||
* <tt>[[Linux Commands#lscpu|lscpu]]</tt>, displaying information about the CPU architecture |
|||
* <tt>[[Linux Commands#inxi|inxi]]</tt>, Command line system information script for console and IRC |
|||
<source lang=bash> |
|||
inxi -Gx # as current X desktop user, no sudo |
|||
</source> |
</source> |
||
* <tt>[[Linux Commands#lsmod / modinfo / modprobe|lsmod]]</tt>, Show the status of modules in the Linux Kernel |
* <tt>[[Linux Commands#lsmod / modinfo / modprobe|lsmod]]</tt>, Show the status of modules in the Linux Kernel |
||
Line 110: | Line 406: | ||
<source lang="bash">modinfo iwlagn</source> |
<source lang="bash">modinfo iwlagn</source> |
||
* <tt>[[Linux Commands#dmidecode|dmidecode]]</tt>, DMI (SMBIOS) table decoder |
* <tt>[[Linux Commands#dmidecode|dmidecode]]</tt>, DMI (SMBIOS) table decoder |
||
:To get BIOS version information: |
|||
<source lang=bash> |
|||
sudo dmidecode | grep "BIOS Information" -A3 |
|||
# BIOS Information |
|||
# Vendor: Hewlett-Packard |
|||
# Version: L71 Ver. 01.30 |
|||
# Release Date: 12/09/2014 |
|||
</source> |
|||
* <tt>[[Linux Commands#uname|uname]]</tt>, print system information |
* <tt>[[Linux Commands#uname|uname]]</tt>, print system information |
||
<source lang="bash">uname -rm</source> |
<source lang="bash">uname -rm</source> |
||
Line 125: | Line 429: | ||
# /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/8p, 480M |
# /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/8p, 480M |
||
</source> |
</source> |
||
:{{red|USB devices are located at <tt>/dev/bus/usb</tt>, according to their bus and device id}} (for instance <tt>/dev/bus/usb/001/002</tt> for bus 001, dev 002) |
|||
* <tt>[[USB|usb-ctrl]]</tt>, control USB power |
|||
* <tt>[[USB|usb-ctrl]]</tt>, control USB power (compiled from [http://www.gniibe.org/oitoite/ac-power-control-by-USB-hub/hub-ctrl.c hub-ctrl.c], see [[USB]]) |
|||
<source lang=bash> |
|||
usb-ctrl -v # Get list of devices and status |
|||
usb-ctrl -b 1 -d 1 -P 1 # Shut down port 1 on bus 1, dev 1 |
|||
</source> |
|||
* <tt>[[Linux Commands#usb-devices|usb-devices]]</tt>, list USB devices |
* <tt>[[Linux Commands#usb-devices|usb-devices]]</tt>, list USB devices |
||
<source lang=bash> |
|||
usb-devices |
|||
# T: Bus=01 Lev=02 Prnt=02 Port=03 Cnt=02 Dev#= 4 Spd=12 MxCh= 0 |
|||
# D: Ver= 2.00 Cls=ff(vend.) Sub=01 Prot=01 MxPS=64 #Cfgs= 1 |
|||
# P: Vendor=413c ProdID=8197 Rev=01.12 |
|||
# S: Manufacturer=Dell Computer Corp |
|||
# S: Product=DW380 Bluetooth Module |
|||
# S: SerialNumber=2016D895569F |
|||
# C: #Ifs= 4 Cfg#= 1 Atr=e0 MxPwr=0mA |
|||
# I: If#= 0 Alt= 0 #EPs= 3 Cls=ff(vend.) Sub=01 Prot=01 Driver=btusb |
|||
# I: If#= 1 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=01 Prot=01 Driver=btusb |
|||
# I: If#= 2 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=ff Prot=ff Driver=(none) |
|||
# I: If#= 3 Alt= 0 #EPs= 0 Cls=fe(app. ) Sub=01 Prot=01 Driver=(none) |
|||
</source> |
|||
* Debug information |
* Debug information |
||
<source lang="bash">dmesg | egrep -i "wlan|iwl"</source> |
<source lang="bash">dmesg | egrep -i "wlan|iwl"</source> |
||
Line 135: | Line 458: | ||
# direct rendering: Yes |
# direct rendering: Yes |
||
</source> |
</source> |
||
* Input devices in <tt>/proc</tt> (some not listed by commands above — e.g. synaptic touchpad): |
|||
* usb-ctrl (compiled from [http://www.gniibe.org/oitoite/ac-power-control-by-USB-hub/hub-ctrl.c hub-ctrl.c], see [[USB]]) |
|||
<source lang=bash> |
<source lang=bash> |
||
cat /proc/bus/input/devices # List input devices |
|||
</source> |
|||
usb-ctrl -b 1 -d 1 -P 1 # Shut down port 1 on bus 1, dev 1 |
|||
* udevadm — Query '''udev''' database |
|||
<source lang=bash> |
|||
udevadm info -q all -n /dev/bus/usb/002/001 # Can query using the device name |
|||
udevadm info -q all -p /sys/bus/usb/devices/usb2 # Or using udev device paths (or symlink) |
|||
udevadm info -a -n /dev/bus/usb/002/040 # Walk up the chain of parent device, and for each, show attributes |
|||
</source> |
|||
* View DHCP lease info on client: |
|||
<source lang=bash> |
|||
cat /var/lib/dhcp/dhclient.leases |
|||
</source> |
|||
=== WiFi === |
|||
See [[Wifi]]. |
|||
=== Firmware === |
|||
* [https://wiki.debian.org/Firmware Firmware - Debian wiki] |
|||
* [https://01.org/linuxgraphics/downloads/firmware Intel firmware] |
|||
* [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/ linux-firmware.git] |
|||
Install package {{deb|firmware-linux}} or {{deb|firmware-linux-nonfree}} to update hardware with latest firmware |
|||
<source lang="bash"> |
|||
sudo apt install firmware-linux-nonfree |
|||
# sudo apt install firmware-linux |
|||
</source> |
|||
Sometimes we get messages like the following when running <code>apt</code> (actually generated by <code>update-initramfs</code>) |
|||
update-initramfs: Generating /boot/initrd.img-5.17.0-1-amd64 |
|||
W: Possible missing firmware /lib/firmware/i915/adlp_dmc_ver2_14.bin for module i915 |
|||
We can find the latest available firmware on the kernel git [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/ linux-firmware.git]. |
|||
To fix the message, copy the firmware to {{file|/lib/firmware}} (see also note below), and reconfigure the package |
|||
<source lang="bash"> |
|||
sudo cp *.bin /lib/firmware/i915 # for i915 firmware |
|||
sudo dpkg-reconfigure linux-image-5.17.0-1-amd64 # Trigger initramfs again |
|||
</source> |
</source> |
||
;Debian |
|||
;Wireless |
|||
(From Debian Wiki) firmware is sourced from the following places (see udev's <code>/lib/udev/hotplug.functions</code> and <code>/lib/udev/firmware.agent</code>): |
|||
* <tt>[[Linux Commands#iwconfig|iwconfig]]</tt>, configure a wireless network interface |
|||
* {{file|/lib/firmware/$(uname -r)}} - Firmware provided by a package, specific for a kernel. |
|||
<source lang="bash">iwconfig wlan0</source> |
|||
* {{file|/lib/firmware/}} - Firmware provided by a package, valid for all kernels. |
|||
* <tt>[[Linux Commands#iwlist|iwlist]]</tt>, get more detailed wireless information from a wireless interface |
|||
* {{file|/usr/local/lib/firmware}} - Location for manually installed firmware. |
|||
<source lang="bash">sudo iwlist scan</source> |
|||
:NOTE: It seems this directory is ignored, we still get the msg about missing firmware, and the message disappear if we copy it in {{file|/lib/firmware/i915}} |
|||
* <tt>[[Linux Commands#rfkill|rfkill]]</tt>, show state of RF SW/HW kill switch (WiFi / BT / ...) |
|||
* {{file|/usr/lib/hotplug/firmware}} - Firmware provided by a package, valid for all kernels |
|||
<source lang="bash">rfkill list all</source> |
|||
=== Wireless firmware === |
=== Wireless firmware === |
||
Line 200: | Line 556: | ||
* or use '''sysinfo''' (<tt>sudo apt-get install sysinfo</tt>), |
* or use '''sysinfo''' (<tt>sudo apt-get install sysinfo</tt>), |
||
* or use '''hardinfo''' (<tt>sudo apt-get install hardinfo</tt>), |
* or use '''hardinfo''' (<tt>sudo apt-get install hardinfo</tt>), |
||
To dump BIOS data: |
|||
* <code>sudo dmidecode</code> |
|||
* <code>sudo dmidecode -t system</code> |
|||
* <code>sudo dmidecode -t system | grep Serial</code>, this returns the serial number label on HP computer for instance. |
|||
=== udev & devfs === |
=== udev & devfs === |
||
Line 310: | Line 671: | ||
options nouveau modeset=0 |
options nouveau modeset=0 |
||
</source> |
</source> |
||
:Note that this will not prevent the X server to load ''Nouveau''. If loaded, this can be unloaded with <code>modprobe -r nouveau</ |
:Note that this will not prevent the X server to load ''Nouveau''. If loaded, this can be unloaded with <code>modprobe -r nouveau</code>, as long as ''Nouveau'' has been prevented from doing a kernel modeset. |
||
My <tt>/etc/X11/xorg.conf</tt> on ''Maverick'': |
My <tt>/etc/X11/xorg.conf</tt> on ''Maverick'': |
||
Line 329: | Line 690: | ||
Option "NoLogo" "True" |
Option "NoLogo" "True" |
||
EndSection |
EndSection |
||
</source> |
|||
=== BlueTooth === |
|||
* Install '''BlueTooth Manager''' to solve bluetooth connection issue (package <tt>blueman</tt>) |
|||
** For instance, solved issues I had with Microsoft Sculpt Mouse (pair device, then set it as '''trusted''', then connect) (see [http://askubuntu.com/questions/253245/bluetooth-mouse-cant-connect-after-reboot], [https://bugs.launchpad.net/ubuntu/+source/bluez/+bug/1094744]) |
|||
=== Udisks === |
|||
The ''[[https://wiki.archlinux.org/index.php/Udev#Udisks udisks]'' daemon serves as an interface to system block devices. It is responsible to mount automatically inserted DVDs. |
|||
;Troubleshooting - Fix broken permissions on UDF |
|||
* Some DVD recorder do not set correctly the directory permissions on DVDs (UDF file system) (missing 'x' flags) |
|||
* There is a fix in udisk that force directory permissions to 0500, but '''only if DVD is read-only''' ([https://bugs.launchpad.net/ubuntu/+source/udisks/+bug/635499 bug 635499], see fix [http://cgit.freedesktop.org/udisks/commit/?id=2d1901f74725da29c7af7602e1c74faf55f14672]) |
|||
* The fix does not work for DVD-RW that are not finalized yet. These discs are then not readable on Linux. |
|||
* There is no way apparently to override the default mount options in ''udisks'' [http://forums.opensuse.org/english/get-technical-help-here/hardware/473592-where-default-permissions-ownership-removable-media-opensuse-12-1-a.html], [http://askubuntu.com/questions/289696/set-mount-options-for-removable-devices-on-kubuntu]). So our only hope is to patch the ''udisks'' package directly. |
|||
* The following is a patch on ''udisk=1.0.4-5ubuntu2.1'' to force dmode=0500 for all optical discs with UDF file systems (to rebuild a package, see [[Package Management|this page]]): |
|||
<source lang=diff> |
|||
diff --git a/src/device.c b/src/device.c |
|||
index a7f8880..3174628 100644 |
|||
--- a/src/device.c |
|||
+++ b/src/device.c |
|||
@@ -6204,10 +6204,8 @@ struct Job |
|||
/* dynamic default options */ |
|||
/* some broken DVDs come with 0400 directory permissions, making them |
|||
- * unreadable; overwrite readonly UDF media with a 0500 dmode. */ |
|||
- if (g_strcmp0 (device->priv->id_type, "udf") == 0 && device->priv->device_is_optical_disc && |
|||
- device->priv->drive_media != NULL && |
|||
- strstr(device->priv->drive_media, "_rw") == NULL && strstr(device->priv->drive_media, "_ram") == NULL) |
|||
+ * unreadable; overwrite all UDF media with a 0500 dmode. */ |
|||
+ if (g_strcmp0 (device->priv->id_type, "udf") == 0 && device->priv->device_is_optical_disc) |
|||
{ |
|||
g_ptr_array_add (options, g_strdup("dmode=0500")); |
|||
} |
|||
</source> |
|||
* Install the fixed package, and check that mount has the new ''dmode=0500'' flag: |
|||
/dev/sr0 on /media/DVD VR type udf (ro,nosuid,nodev,uid=6659,gid=6659,iocharset=utf8,umask=0077,dmode=0500,uhelper=udisks) |
|||
* Alternate patch. This force flag ''dmode=0500'' for all UDF file system, similar to iso9660: |
|||
<source lang=diff> |
|||
diff --git a/src/device.c b/src/device.c |
|||
index 2d7621b..a7f8880 100644 |
|||
--- a/src/device.c |
|||
+++ b/src/device.c |
|||
@@ -5926,7 +5926,7 @@ struct Job |
|||
/* ---------------------- udf -------------------- */ |
|||
-static const char *udf_defaults[] = { "uid=", "gid=", "iocharset=utf8", "umask=0077", "dmode=0500", NULL }; |
|||
+static const char *udf_defaults[] = { "uid=", "gid=", "iocharset=utf8", "umask=0077", NULL }; |
|||
static const char *udf_allow[] = { "iocharset=", "umask=", "mode=", "dmode=", NULL }; |
|||
static const char *udf_allow_uid_self[] = { "uid=", NULL }; |
|||
static const char *udf_allow_gid_self[] = { "gid=", NULL }; |
|||
@@ -6203,6 +6203,15 @@ struct Job |
|||
/* dynamic default options */ |
|||
+ /* some broken DVDs come with 0400 directory permissions, making them |
|||
+ * unreadable; overwrite readonly UDF media with a 0500 dmode. */ |
|||
+ if (g_strcmp0 (device->priv->id_type, "udf") == 0 && device->priv->device_is_optical_disc && |
|||
+ device->priv->drive_media != NULL && |
|||
+ strstr(device->priv->drive_media, "_rw") == NULL && strstr(device->priv->drive_media, "_ram") == NULL) |
|||
+ { |
|||
+ g_ptr_array_add (options, g_strdup("dmode=0500")); |
|||
+ } |
|||
+ |
|||
/* user supplied options */ |
|||
for (n = 0; given_options[n] != NULL; n++) |
|||
{ |
|||
</source> |
|||
For '''udisk2''': |
|||
<source lang=diff> |
|||
diff --git a/src/udiskslinuxfilesystem.c b/src/udiskslinuxfilesystem.c |
|||
-static const gchar *udf_defaults[] = { "uid=", "gid=", "iocharset=utf8", "umask=0077", NULL }; |
|||
-static const gchar *udf_allow[] = { "iocharset=", "umask=", NULL }; |
|||
+static const gchar *udf_defaults[] = { "uid=", "gid=", "iocharset=utf8", "umask=0077", "dmode=0500", NULL }; |
|||
+static const gchar *udf_allow[] = { "iocharset=", "umask=", "dmode=", NULL }; |
|||
</source> |
</source> |
||
Line 365: | Line 802: | ||
ldconfig -n directory_with_shared_libraries #Creates soname links to realname when installing new libraries |
ldconfig -n directory_with_shared_libraries #Creates soname links to realname when installing new libraries |
||
ldd /bin/ls #List shared libraries needed by a given executable |
ldd /bin/ls #List shared libraries needed by a given executable |
||
</source> |
|||
;List symbols exported by libraries: |
|||
From [http://stackoverflow.com/questions/34732/how-do-i-list-the-symbols-in-a-so-file SO]: |
|||
<source lang="bash"> |
|||
nm -g yourLib.so # Standard tool |
|||
nm -gC yourLib.so # ... C++ lib, demangle the symbols |
|||
objdump -TC libz.so # For library in ELF format - -C to demangle |
|||
readelf -Ws libz.so # ... Another solution for ELF |
|||
readelf -a /some/exec |
|||
</source> |
</source> |
||
Line 377: | Line 824: | ||
</source> |
</source> |
||
== |
=== Troubleshooting === |
||
==== Use LD_PRELOAD to fix <code>symbol '...': can't resolve symbol</code> ==== |
|||
=== Commands === |
|||
<source lang=bash> |
|||
See [[#Hardware|Hardware section above]]. |
|||
cat bup_intl_issue.py |
|||
# import posix1e; # From package pylibacl |
|||
=== Linux Access Point === |
|||
python bup_intl_issue.py |
|||
You can turn your laptop into a WiFi access point, provided of course that your WiFi card supports that feature: |
|||
# /opt/bin/python2.7: symbol 'libintl_gettext': can't resolve symbol |
|||
# Traceback (most recent call last): |
|||
# File "bup_intl_issue.py", line 1, in <module> |
|||
# import posix1e; |
|||
# ImportError: unknown dlopen() error |
|||
</source> |
|||
We can fix the issue using the <code>LD_PRELOAD</code> trick though: |
|||
<source lang="bash"> |
<source lang="bash"> |
||
LD_PRELOAD=/opt/lib/libintl.so python2.7 bup_intl_issue.py |
|||
sudo service network-manager stop |
|||
sudo iwconfig wlan0 mode master |
|||
</source> |
</source> |
||
If you get the following error then most probably it is not supported: |
|||
==== <code>No such file or directory</code> when executing a program ==== |
|||
<source lang="text"> |
|||
Bash returns an error <code>No such file or directory</code> even though the executable is there: |
|||
Error for wireless request "Set Mode" (8B06) : |
|||
SET failed on device wlan0 ; Invalid argument. |
|||
<source lang=bash> |
|||
./lmgrd |
|||
# bash ./lmgrd: No such file or directory |
|||
ls -l ./lmgrd |
|||
# -rwxr-xr-x 1 superman superman 1.5 Nov 21 2016 ./lmgrd |
|||
</source> |
</source> |
||
Turn NetworkManager back on: |
|||
The problem is that the executable requires a library that is not present (likely the interpreter): |
|||
<source lang="bash"> |
<source lang="bash"> |
||
readelf -a ./lmgrd | grep -i interpreter # Could also do 'ldd ./lmgrd' |
|||
sudo service network-manager start |
|||
# [Requesting program interpreter: ./lib64/ld-lsb-x86-64.so.3] |
|||
cd /lib64 |
|||
sudo ln -sf /lib/x86_64-linux-gnu/ld-2.19.so ld-lsb-x86-64.so.3 |
|||
</source> |
</source> |
||
==== Troubleshooting process using libusb ==== |
|||
Some links: |
|||
* [https://help.ubuntu.com/community/WifiDocs/MasterMode MasterMode on Ubuntu Wiki] |
|||
<code>strace</code> is the tool of choice. |
|||
* [https://help.ubuntu.com/community/WifiDocs/WirelessAccessPoint WirelessAccessPoint on Ubuntu Wiki] |
|||
* [http://www.devshed.com/c/a/BrainDump/Building-a-Linux-Wireless-Access-Point/ Building a Linux Wireless Access Point] |
|||
<source lang="bash"> |
|||
strace someprocess &> strace.txt |
|||
</source> |
|||
* We look for the application error messages in strace.txt. |
|||
* Look at system calls that happened before that error messages. Is the errror due to bad permission, or device busy? |
|||
:* Bad permission, check device permission (usually {{file|/dev/bus/usb}}). Probably the device needs a udev rule to setup the permission. |
|||
:* Device busy, check for which module is grapping the device. Maybe this device must be black-listed in {{file|/etc/modprobe.d}}. |
|||
* Look which libraries are loaded (calls to <code>open</code>). Are all libraries found? |
|||
It might be necessary to follow child processes. Use <code>stace -f</code> for this. |
|||
Another useful tool is <code>strace</code> (to trace calls to dynamically-linked libraries). |
|||
== Network == |
|||
See also [[Linux networking]] and [[Wifi]]. |
|||
=== Commands === |
|||
See [[#Hardware|Hardware section above]]. |
|||
=== ZeroConfig === |
=== ZeroConfig === |
||
ZeroConfig refers to all utilities that help setting up network '''without''' any additional configuration. |
ZeroConfig refers to all utilities that help setting up network '''without''' any additional configuration. Somes links: |
||
* [http://en.wikipedia.org/wiki/Zero_configuration_networking wikipedia] |
|||
* [https://wiki.debian.org/ZeroConf ZeroConf] on debian.org. |
|||
* [http://www.zeroconf.org/ zeroconf.org] website (open standard issued by Apple). |
|||
To install on Linux (Ubuntu): |
|||
<source lang=bash> |
|||
sudo apt-get install libnss-mdns avahi-daemon mdns-scan |
|||
</source> |
|||
;Address resolution |
;Address resolution |
||
:To be completed. |
|||
;Name resolution |
;Name resolution |
||
* By default, any given host can be accessed via the name <code>hostname.local</code> without need of a local DNS server. |
|||
<source lang=bash> |
|||
ping myserver.local |
|||
</source> |
|||
* Name resolution relies on '''mDNS''' (multicast DNS) protocol. mDNS client makes a request to a well-known multicast address (<tt>224.0.0.251</tt> for IPv4 and <tt>ff02::fb</tt> for IPv6 link-local addressing). On Linux, '''avahi''' package implements the Apple Zeroconf specification. |
|||
* To list all names that are broadcasted: |
|||
:On Linux, '''avahi''' package implements the Apple Zeroconf specification. |
|||
<source lang=bash> |
|||
mdns-scan |
|||
</source> |
|||
;Service discovery |
;Service discovery |
||
:To be completed? |
|||
=== Network Manager - Search Path === |
=== Network Manager - Search Path === |
||
Line 439: | Line 938: | ||
* In ''Control Panel'' → ''Internet Options'' → ''Content'' tab, click '''Certificates''', or<br/>alternatively, type {{kb|Win-R}}, <tt>certmgr.msc</tt>. |
* In ''Control Panel'' → ''Internet Options'' → ''Content'' tab, click '''Certificates''', or<br/>alternatively, type {{kb|Win-R}}, <tt>certmgr.msc</tt>. |
||
* In the ''Personal'' tab, select the certificate used for ''client authentication'', and click '''Export'''. |
* In the ''Personal'' tab, select the certificate used for ''client authentication'', and click '''Export'''. |
||
* In the new window, click ''Next'', then select '''Yes, export the private key''' and click ''Next''. |
* In the new window, click ''Next'', then select '''Yes, export the private key''' and click ''Next''. |
||
: If the option ''export the private key'' is greyed out, see [[Windows Administration]]. |
|||
* Select format ''Personal information interchange - PKCS #12 (.PFX)'', and select ''Include all the certificates in the certificate path if possible'' and ''Enable strong protection''. |
* Select format ''Personal information interchange - PKCS #12 (.PFX)'', and select ''Include all the certificates in the certificate path if possible'' and ''Enable strong protection''. |
||
* Select a password, and save the file (say <tt>mywindowscert.pfx</tt>). |
* Select a password, and save the file (say <tt>mywindowscert.pfx</tt>). |
||
<li>In Ubuntu, '''split the exported certificate''' in the components '''CA / Cert / Private key''' |
<li>In Ubuntu, '''split the exported certificate''' in the components '''CA / Cert / Private key'''.</li> |
||
* See [[SSL]]. |
|||
<source lang="bash"> |
|||
openssl pkcs12 -in mywindowscert.pfx -nocerts -out mycert.key.p12 |
|||
openssl pkcs12 -in mywindowscert.pfx -clcerts -nokeys -out mycert.crt.pem |
|||
openssl pkcs12 -in mywindowscert.pfx -cacerts -nokeys -out mycert.ca.pem |
|||
</source> |
|||
<li>Now create a new wireless network connection in Ubuntu:</li> |
<li>Now create a new wireless network connection in Ubuntu:</li> |
||
* Security: '''Dynamic WEP (802.1x)''' or '''WPA & WPA2 Enterprise''' |
* Security: '''Dynamic WEP (802.1x)''' or '''WPA & WPA2 Enterprise''' |
||
Line 457: | Line 953: | ||
* Private key password: as required |
* Private key password: as required |
||
</ol> |
</ol> |
||
More information: |
|||
* https://wiki.freeradius.org/protocol/EAP-PEAP#types_peap-eap-tls |
|||
* https://en.wikipedia.org/wiki/Protected_Extensible_Authentication_Protocol |
|||
=== Firewall === |
=== Firewall === |
||
Line 469: | Line 969: | ||
netstat -ln > netstat-ln-smb.after |
netstat -ln > netstat-ln-smb.after |
||
diff -u netstat-ln-smb.* |
diff -u netstat-ln-smb.* |
||
</source> |
|||
=== Network Time Protocol (ntp) === |
|||
See [[ntp]]. |
|||
==Chroot== |
|||
See [[Chroot]] page. |
|||
== TTY == |
|||
Stuff to read some day: |
|||
* [http://linusakesson.net/programming/tty/index.php The TTY Demystified] |
|||
* [https://stackoverflow.com/questions/6108953/how-does-ctrl-c-terminate-a-child-process Stack Overflow - How does Ctrl-C terminate a child process] |
|||
* [https://unix.stackexchange.com/questions/149741/why-is-sigint-not-propagated-to-child-process-when-sent-to-its-parent-process Why is SIGINT not propagated to child process when sent to its parent process] |
|||
== Printing == |
|||
=== CUPS === |
|||
* Current print server on Ubuntu is '''CUPS''' |
|||
* '''Web Interface''' — CUPS server is accessed via at http://localhost:631 (see [https://help.ubuntu.com/lts/serverguide/cups.html CUPS-Print Server]) |
|||
;Reset CUPS printer |
|||
:Go to '''http://localhost:631''' to reset CUPS printer |
|||
;More troubleshooting |
|||
:See [https://wiki.ubuntu.com/DebuggingPrintingProblems] |
|||
;Setup backend error handler |
|||
*See [http://ubuntu.5.x6.nabble.com/cups-restart-printer-with-scipt-td1679898.html], [http://www.linuxfoundation.org/collaborate/workgroups/openprinting/database/backenderrorhandler] |
|||
*''beh'' is already installed on Ubuntu 12.04+. |
|||
*Another solution is to edit the policy ([http://superuser.com/questions/280396/how-to-resume-cups-printer-from-command-line]). |
|||
*Default policy is ''retry-job'', but I still get error message ''file is rejected''. I tried then ''abort-job''. |
|||
;Command line interface |
|||
<source lang="bash"> |
|||
# Get printers status |
|||
lpstat -p |
|||
# printer Canon_MG5300_series is idle. enabled since Wed 25 Dec 2019 06:52:57 PM CET |
|||
# printer DESKJET-940C is idle. enabled since Wed 25 Dec 2019 06:50:32 PM CET |
|||
# printer TS8100 is idle. enabled since Tue 13 Aug 2019 04:22:10 PM CEST |
|||
# printer Canon_MG5300_series disabled since Wed 25 Dec 2019 06:52:37 PM CET - |
|||
# Paused |
|||
# printer DESKJET-940C is idle. enabled since Wed 25 Dec 2019 06:50:32 PM CET |
|||
# printer TS8100 is idle. enabled since Tue 13 Aug 2019 04:22:10 PM CEST |
|||
# printer Canon_MG5300_series is idle. enabled since Wed 25 Dec 2019 06:13:46 PM CET |
|||
# printer DESKJET-940C now printing DESKJET-940C-288. enabled since Wed 25 Dec 2019 06:47:01 PM CET |
|||
# Processing page 2... |
|||
# printer TS8100 is idle. enabled since Tue 13 Aug 2019 04:22:10 PM CEST |
|||
# Enable paused printer |
|||
cupsenable DESKJET-940C |
|||
</source> |
</source> |
||
Line 487: | Line 1,039: | ||
* Add <tt>--verbose</tt> |
* Add <tt>--verbose</tt> |
||
=== Alt-SysRq-REISUB / Alt-PrtScr-REISUB === |
|||
== <tt>/var/log</tt> == |
|||
Using the [http://en.wikipedia.org/wiki/Magic_SysRq_key Magic SysRq key], one can usually reboot its system nicely (better than holding the power button for 5sec). |
|||
Press and hold {{kb|Alt-SysRq}}, then press the following keys in sequence, waiting ''1 second'' between each press: {{kb|R}}, {{kb|E}}, {{kb|I}},{{kb|S}}, {{kb|U}}, {{kb|B}}. |
|||
On laptop, you often have to press and hold {{kb|Alt}}, then {{kb|Fn-SysRq}}, then release {{kb|Fn}} while holding {{kb|Alt-SysRq}}, and finally {{kb|R E I S U B}}. |
|||
On keyboard without {{kb|SysRq}}, the combination {{kb|Alt-PrtScr}} also works. |
|||
un'''R'''aw (take control of keyboard back from X), |
|||
t'''E'''rminate (send SIGTERM to all processes, allowing them to terminate gracefully), |
|||
k'''I'''ll (send SIGKILL to all processes, forcing them to terminate immediately), |
|||
'''S'''ync (flush data to disk), |
|||
'''U'''nmount (remount all filesystems read-only), |
|||
re'''B'''oot. |
|||
A mnemonic "'''R'''eboot '''E'''ven '''I'''f '''S'''ystem '''U'''tterly '''B'''roken", or '''BUSIER''' when read backwards |
|||
=== <code>/proc/sysrq-trigger</code> === |
|||
SysRq can also be triggered by writting to {{file|/proc/sysrq-trigger}} [https://www.kernel.org/doc/html/latest/admin-guide/sysrq.html]: |
|||
<source lang="bash"> |
|||
echo u | /proc/sysrq-trigger # Remount all fs read-only |
|||
</source> |
|||
== Troubleshooting == |
|||
References: |
|||
* [https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks KernelDebuggingTricks] (wiki.ubuntu.com) |
|||
=== Serial Console === |
|||
References: |
|||
* [https://help.ubuntu.com/community/SerialConsoleHowto SerialConsoleHowto] (help.ubuntu.com) |
|||
* [https://www.virtualbox.org/wiki/Serial_redirect How to redirect the Linux guest kernel console to a serial port / file] (www.virtualbox.org) |
|||
* [[http://www.gonwan.com/2014/04/07/setting-up-serial-console-on-virtualbox/ Setting up serial console on VirtualBox] |
|||
Use the ''serial console'' to debug kernel freezes, hangs... that leaves no traces in the kernel logs. |
|||
To enable the serial console, you need to have a serial port (named {{file|/dev/ttyS0}} on Linux, COM1 on windows), and have a process like {{deb|minicom}} that connects to that port on the machine to debug. |
|||
To enable, edit {{file|/etc/default/grub}}, and add the line: |
|||
<source lang=diff> |
|||
-GRUB_CMDLINE_LINUX_DEFAULT="splash quiet" |
|||
+GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0 console=tty0 ignore_loglevel" |
|||
</source> |
|||
The first parameter will ensure that all kernel output is redirected to the serial console. The second parameter will ensure that the kernel output is still logged to the guest text console. The third parameter enforces the guest Linux kernel to print all kernel messages to the console. |
|||
Update grub and restart. |
|||
<source lang=bash> |
|||
sudo update-grub |
|||
sudo shutdown -r now |
|||
</source> |
|||
;VirtualBox |
|||
On '''VirtualBox''' [https://www.virtualbox.org/wiki/Serial_redirect], one can easily configure a virtual serial port. In the settings, open the ''Serial Ports'' tab: |
|||
* Enable serial port '''COM1''' (irq 4, io 0x3F8). |
|||
* Select mode '''Raw File''', and select a file on the host file system. |
|||
Restart the guest, and verify that the port is running: |
|||
<source lang=bash> |
|||
sudo stty -F /dev/ttyS0 -a |
|||
# speed 9600 baud; rows 0; columns 0; line = 0; |
|||
# intr = ^C; quit = ^; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; |
|||
# eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; |
|||
# werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0; |
|||
</source> |
|||
Another option is to use ''Host pipe'', and use for instance minicom to attach to the named pipe [http://www.gonwan.com/2014/04/07/setting-up-serial-console-on-virtualbox/], [http://wiki.illumos.org/display/illumos/Serial+Console+in+VirtualBox], [https://gist.github.com/snb/284940/11e6354f170be602c9c2f67b59d489ed49ebd143]. |
|||
=== Netconsole === |
|||
References: |
|||
* [https://wiki.ubuntu.com/Kernel/Netconsole Ubuntu Netconsole] |
|||
* See [https://michael.stapelberg.ch/posts/2013-09-16-remote_syslog/ Remote kernel logs with netconsole and rsyslog (2013)] |
|||
* [http://www.cyberciti.biz/files/linux-kernel/Documentation/networking/netconsole.txt netconsole documentation] |
|||
* '''dmesg(8)''' man page and {{file|Documentation/kernel-parameters.txt}}. |
|||
''netconsole'' is similar to the serial console, except that messages are sent to a network socket instead of a serial link. |
|||
;Setup Receiver side |
|||
On the remote host, the simpler is to start a netcat process or equivalent to collect the log: |
|||
<source lang="bash"> |
|||
nc -l -p 6666 -u # ... OR ... |
|||
netcat -u -l -p 6666 # ... OR ... |
|||
nc -l -p 6666 -u > netconsole.txt & |
|||
</source> |
|||
A more stable solution is to use <code>syslog-ng</code> (see Ubuntu wiki above) or <code>rsyslog</code> (see Stapelberg's link above). |
|||
To setup rsyslog: |
|||
* First create the log folder and rsyslog config: |
|||
<source lang="bash"> |
|||
# Here we assume sender hostname is 'zavcxl0006' with IP 192.168.1.4 |
|||
sudo mkdir -p /var/log/remote/zavcxl0006 |
|||
cat >/etc/rsyslog.d/remote.conf <<EOT |
|||
\$ModLoad imudp |
|||
\$RuleSet remote |
|||
# For each IP address that you want to store logs from, |
|||
# add and modify the following two (!) lines: |
|||
if \$fromhost-ip=='192.168.1.4' then /var/log/remote/zavcxl0006/console.log |
|||
& stop |
|||
\$InputUDPServerBindRuleset remote |
|||
\$UDPServerRun 6666 |
|||
\$RuleSet RSYSLOG_DefaultRuleset |
|||
EOT |
|||
</source> |
|||
* Create the logrotate config: |
|||
<source lang="bash"> |
|||
cat >/etc/logrotate.d/remote <<'EOT' |
|||
/var/log/remote/*/*.log |
|||
{ |
|||
copytruncate |
|||
rotate 30 |
|||
daily |
|||
missingok |
|||
dateext |
|||
notifempty |
|||
delaycompress |
|||
compress |
|||
maxage 31 |
|||
postrotate |
|||
invoke-rc.d rsyslog reload > /dev/null |
|||
endscript |
|||
} |
|||
EOT |
|||
</source> |
|||
;Setup Sender side |
|||
To use it, first install it from debian packages: |
|||
<source lang="bash"> |
|||
sudo apt install netconsole |
|||
</source> |
|||
To enable netconsole you must change kernel option at boot time. Edit {{file|/etc/default/grub}}: |
|||
<source lang="diff"> |
|||
-GRUB_CMDLINE_LINUX_DEFAULT="splash quiet" |
|||
+GRUB_CMDLINE_LINUX_DEFAULT="debug ignore_loglevel" |
|||
</source> |
|||
Then update grub: |
|||
<source lang="bash"> |
|||
sudo update-grub |
|||
</source> |
|||
Or set it via <code>dmesg</code> (see '''dmesg(8)''' man page and {{file|Documentation/kernel-parameters.txt}} for details) |
|||
netconsole can be started automatically with a static configuration. To start at boot time, update {{file|/etc/modules}} and create a config file in {{file|/etc/modprobe.d}}: |
|||
<source lang="bash"> |
|||
sudo sh -c 'echo netconsole >> /etc/modules' |
|||
sudo sh -c 'echo options netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82 > /etc/modprobe.d/netconsole.conf' |
|||
</source> |
|||
Or start it at command-line: |
|||
<source lang="bash"> |
|||
sudo modprobe netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82 |
|||
</source> |
|||
The syntax is: |
|||
<source lang="bash"> |
|||
netconsole="[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr][;...]" |
|||
# src-port source for UDP packets (defaults to 6665) |
|||
# src-ip source IP to use (interface address) - MANDATORY at boot time |
|||
# dev network interface (eth0) |
|||
# tgt-port port for logging agent (6666) |
|||
# tgt-ip IP address for logging agent |
|||
# tgt-macaddr ethernet MAC address for logging agent (broadcast) |
|||
</source> |
|||
Quotes are mandatory for multi-host configuration (semi-colon) but optional otherwise. Giving the ''src-ip'' is mandatory at boot time (when starting module in {{file|/etc/modules}} because the interface does not have an IP address yet). |
|||
netconsole can also be configured dynamically [http://www.cyberciti.biz/files/linux-kernel/Documentation/networking/netconsole.txt] using ''configfs'' [http://www.linux.org/threads/sysfs-and-configfs.4956/]: |
|||
<source lang="bash"> |
|||
sudo mount -t configfs none /sys/kernel/config |
|||
cd /sys/kernel/config |
|||
sudo mkdir target_a |
|||
cd target_a |
|||
cat enabled # check if enabled is 1 |
|||
echo 0 > enabled # disable the target (if required) |
|||
echo eth2 > dev_name # set local interface |
|||
echo 10.0.0.4 > remote_ip # update some parameter |
|||
echo cb:a9:87:65:43:21 > remote_mac # update more parameters |
|||
echo 1 > enabled # enable target again |
|||
</source> |
|||
Note netconsole module cannot be removed if configfs contains a dynamic target. |
|||
Alternatively, here a script to configure dynamically <code>netconsole</code> module: |
|||
<source lang="bash"> |
|||
unset NETCON_TGTHOST |
|||
eval $(get-network.sh < ~/etc/network_definition) |
|||
case $LOCATION in |
|||
home) |
|||
# Require |
|||
# ipkg install netcat |
|||
# mkdir -p /shares/beq06659/netconsole |
|||
# nc -l -p 6666 -u >/shares/beq06659/netconsole/zavcxl0005-netconsole& |
|||
NETCON_TGTHOST=lacie-cloudbox |
|||
;; |
|||
stzav) |
|||
NETCON_TGTHOST=apple-pi.zav.st.com |
|||
;; |
|||
esac |
|||
echo "Resetting netconsole for location '$LOCATION'..." |
|||
sudo rmmod netconsole 2> /dev/null |
|||
if [ -n "$NETCON_TGTHOST" ]; then |
|||
NETCON_PORT=6666 |
|||
# Adding -v -v to nc to avoid it to block sometimes. Go figure... |
|||
echo "[------------] Redocking on $(date +"%Y-%m-%d %H:%M:%S")..." | nc -v -v -u -p 6665 $NETCON_TGTHOST $NETCON_PORT |
|||
NETCON_TGTIP=$(getent hosts $NETCON_TGTHOST|awk '{print $1}') |
|||
ping -c 1 $NETCON_TGTIP >/dev/null |
|||
NETCON_TGTMAC=$(arp -n $NETCON_TGTIP|awk '/ether/{print $3}') |
|||
echo "... setting up netconsole for location '$LOCATION' ($NETCON_TGTHOST,$NETCON_TGTIP/$NETCON_TGTMAC)" |
|||
sudo modprobe netconsole netconsole=@/,$NETCON_PORT@$NETCON_TGTIP/$NETCON_TGTMAC |
|||
else |
|||
echo "... no target available for netconsole at location $LOCATION" |
|||
fi |
|||
</source> |
|||
On Debian, netconsole can be setup more easily using <code>netconsole-setup</code>: |
|||
<source lang="bash"> |
|||
echo "Resetting netconsole for location '$LOCATION'..." |
|||
sudo rmmod netconsole 2> /dev/null |
|||
if [ -n "$NETCON_TGTHOST" ]; then |
|||
NETCON_PORT=6666 |
|||
# Adding -v -v to nc to avoid it to block sometimes. Go figure... |
|||
echo "[------------] Redocking on $(date +"%Y-%m-%d %H:%M:%S")..." | nc -v -v -w 1 -u -p 6665 $NETCON_TGTHOST $NETCON_PORT |
|||
echo "... using target host '$NETCON_TGTHOST'." |
|||
sudo netconsole-setup 6666@$NETCON_TGTHOST |
|||
else |
|||
echo "... no target available for netconsole at location $LOCATION." |
|||
fi |
|||
</source> |
|||
;Troubleshooting |
|||
* If getting message |
|||
$ sudo modprobe netconsole netconsole=@/,6666@192.168.1.3/b8:27:eb:69:3e:df |
|||
modprobe: ERROR: could not insert 'netconsole': No such device |
|||
:Try to add the name of the network interface to use (see [https://unix.stackexchange.com/questions/304788/linux-netconsole-with-bridged-interface]). |
|||
== System Logs == |
|||
=== Using systemd === |
|||
To view kernel logs (with highlighting, grouping...): |
|||
<source lang="bash"> |
|||
sudo journalctl -b # Since last boot |
|||
# or |
|||
sudo journalctl -fe # Last event and follow |
|||
</source> |
|||
=== {{file|/var/log}} === |
|||
Some external links: |
Some external links: |
||
* [http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch05_:_Troubleshooting_Linux_with_syslog#How_to_View_New_Log_Entries_as_They_Happen Quick HOWTO - Troubleshooting Linux with syslog] |
* [http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch05_:_Troubleshooting_Linux_with_syslog#How_to_View_New_Log_Entries_as_They_Happen Quick HOWTO - Troubleshooting Linux with syslog] |
||
* [http://library.gnome.org/users/gnome-system-log/|Gnome System Log Viewer] (package |
* [http://library.gnome.org/users/gnome-system-log/|Gnome System Log Viewer] (package {{deb|gnome-utils}}), is an handy tool to view log files under Gnome, with entries sorted by date, and new entries highlighted in bold. |
||
* '''dmesg(8)''' man page and {{file|Documentation/kernel-parameters.txt}}. |
|||
''syslog'' is an utility to log all system messages, from information messages to critical errors. Log files are stored in |
''syslog'' is an utility to log all system messages, from information messages to critical errors. Log files are stored in {{file|/var/log}}. On Ubuntu, the default logging system is '''rsyslog''', with configuration files {{file|/etc/rsyslog.conf}} and in {{file|/etc/rsyslog.conf.d/}}. |
||
Logs generated by rsyslog (see |
Logs generated by rsyslog (see {{file|/etc/rsyslog.d/50-default.conf}}): |
||
{| class="wikitable" |
{| class="wikitable" |
||
Line 501: | Line 1,305: | ||
!file!!source!!description |
!file!!source!!description |
||
|- |
|- |
||
| |
|{{file|aptitude}}|| || |
||
|- |
|- |
||
| |
|{{file|auth.log}}||rsyslog||Messages to facilities ''auth'' and ''authpriv'' |
||
|- |
|- |
||
| |
|{{file|boot}}|| || |
||
|- |
|- |
||
| |
|{{file|boot.log}}|| || |
||
|- |
|- |
||
| |
|{{file|btmp}}|| || |
||
|- |
|- |
||
| |
|{{file|daemon.log}}||rsyslog||Messages to facility ''daemon'' |
||
|- |
|- |
||
| |
|{{file|debug}}||rsyslog||Messages with ''debug'' priority, but excluding facilities ''auth'', ''mail'' and ''news'' |
||
|- |
|- |
||
|''' |
|'''{{file|dmesg}}'''||'''kernel'''||'''Boot time hardware detection and driver setup (i.e. kernel messages before syslog daemon is launched).'''<br/>This is *not* the same as <code>dmesg</code> output (see {{file|kern.log}})! |
||
|- |
|- |
||
| |
|{{file|dpkg.log}}|| || |
||
|- |
|- |
||
| |
|{{file|faillog}}|| || |
||
|- |
|- |
||
| |
|{{file|fontconfig.log}}|| || |
||
|- |
|- |
||
| |
|{{file|jockey.log}}|| || |
||
|- |
|- |
||
|'''{{file|kern.log}}'''||rsyslog||'''Messages to facility ''kern''''' (apparently <code>dmesg</code> will display the last 16392 octets of {{file|/var/log/kern.log}} since last boot [http://stackoverflow.com/questions/11412950/difference-between-dmesg-and-var-log-kern-log]) |
|||
|<tt>kern.log</tt>||rsyslog||Messages to facility ''kern'' |
|||
|- |
|- |
||
| |
|{{file|lastlog}}||lastlog||last login of each user ([http://linux.die.net/man/8/lastlog]). It looks big, but it's a sparse file (<code>du -h lastlog</code>) !!! |
||
|- |
|- |
||
| |
|{{file|lpr.log}}||rsyslog||Messages to facility ''lpr'' |
||
|- |
|- |
||
| |
|{{file|mail.info}}||rsyslog||Messages to facility ''mail'', priority ≥ ''info'' |
||
|- |
|- |
||
| |
|{{file|mail.err}}||rsyslog||Messages to facility ''mail'', priority ≥ ''err'' |
||
|- |
|- |
||
| |
|{{file|mail.log}}||rsyslog||Messages to facility ''mail'' |
||
|- |
|- |
||
| |
|{{file|mail.warn}}||rsyslog||Messages to facility ''mail'', priority ≥ ''warn'' |
||
|- |
|- |
||
| |
|{{file|messages}}||rsyslog||Messages with ''info'',''notice'' and ''warn'' priority, but excluding facilities ''auth'', ''daemon'', ''mail'' and ''news'' |
||
|- |
|- |
||
| |
|{{file|MountManager.log}}|| || |
||
|- |
|- |
||
| |
|{{file|mysql.err}}|| || |
||
|- |
|- |
||
| |
|{{file|mysql.log}}|| || |
||
|- |
|- |
||
| |
|{{file|pm-powersave.log}}|| || |
||
|- |
|- |
||
| |
|{{file|pm-suspend.log}}|| || |
||
|- |
|- |
||
| |
|{{file|pycentral.log}}|| || |
||
|- |
|- |
||
|''' |
|'''{{file|syslog.log}}'''||'''rsyslog'''||'''All messages except those in {{file|auth.log}} (i.e. facilities ''auth'' and ''authpriv'')''' |
||
|- |
|- |
||
| |
|{{file|udev}}|| || |
||
|- |
|- |
||
| |
|{{file|ufw.log}}||rsyslog||All messages from ''UFW'' firewall |
||
|- |
|- |
||
| |
|{{file|user.log}}||rsyslog||All messages targeting facility ''user'' |
||
|- |
|- |
||
| |
|{{file|vbox-install.log}}|| || |
||
|- |
|- |
||
| |
|{{file|wtmp}}|| || |
||
|- |
|- |
||
| |
|{{file|Xorg.0.log}}|| || |
||
|- |
|- |
||
| |
|{{file|Xorg.failsafe.log}}|| || |
||
|} |
|} |
Latest revision as of 06:05, 5 July 2024
Documentation / Getting Help
- yelp
- Default Gnome help system. Contains basic documentation, manpages, and guides (which can even install applications if clicked on)
- doc-base
- The doc-base package implements a flexible mechanism for handling and presenting documentation. See doc-base on debian.org.
- dwww
- dwww is the web base documentation reader. When installed, you can browse the documentation available on your machine by opening your browser at http://localhost/dwww/. dwww has also command-line support.
Kernel
Architecture (32/64-bit)
32-bit executables can still run on 64-bit architecture (amd-64). Check package ia32-libs.
Note that 32-bit libraries are located in /usr/lib32 and not in /usr/lib
OOM Score (Out of Memory)
Kernel has an advanced algorithm to detect which process to kill when an Out of Memory occures (from [1]):
[...] But, actually, Linux doesn't just pick the process with the failed allocation to kill. Instead, when a process makes a memory request which cannot be fulfilled, the OS runs a quick calculation of the memory usage "badness" of all processes. The base of the badness score is the processes resident memory, plus the resident memory of child processes. Processes that have been "nice'd" get a score boost (on the theory they're likely to be less important), but long-running processes get a score decrease (on the theory they're likely to be more important). Superuser processes have their score decreased. Finally, processes have their scores decreased by a user-settable value in /proc//oom_adj (default is no adjustment). Also, if /proc//oom_adj is set to the constant OOM_DISABLE, then the process is not killable.
When memory runs out, Linux kills the process with the highest score. If a single ordinary user process, especially a short-lived desktop process, has consumed nearly all of the system RAM, and no one has messed with oom_adj for that process, then it WILL be the one that dies. [...]
The OOM score of each process can be obtained with:
find /proc -maxdepth 2 -name oom_score | while read i; do echo -n "$i "; cat $i; done | sort -n -k2
/etc/sudoers
The man page gives a complete but unclear description of the file specification. Here a simplified but complete version:
First the description of possible entries in the file:
# Alias
'User_Alias' NAME '=' User... (':' NAME '=' User... )*
'Runas_Alias' NAME '=' Runas_Member... (':' NAME '=' Runas_Member...)*
'Host_Alias' NAME '=' Host... (':' NAME '=' Host... )*
'Cmnd_Alias' NAME '=' Cmnd... (':' NAME '=' Cmnd... )*
#Default_Entry
'Defaults' ('@' Host... | ':' User... | '!' Cmnd... | '>' Runas_Member...)? Parameter...
#User_Spec
User... Host... '=' Cmnd_Spec... (':' Host... '=' Cmnd_Spec...)*
Now the description of the syntactical elements (note the description of ...
, which is simply a comma-separated list):
identifier... ::= identifier (',' identifier)*
NAME ::= [A-Z]([a-z][A-Z][0-9]_)*
User /
Runas_Member ::= '!'* ( username | '#'uid | '%'group | '+'netgroup | Alias | 'ALL' )
Host ::= '!'* ( hostname | ip_addr | network(/netmask)? | '+'netgroup | Alias| 'ALL' )
Cmnd ::= '!'* ( command filename (args | '""')? | directory | "sudoedit" | Alias | 'ALL' )
Parameter ::= Parameter '=' Value | Parameter '+=' Value | Parameter '-=' Value | '!'* Parameter
Cmnd_Spec ::= ('(' Runas_Member...? (':' ...? ')')? ('NOPASSWD:'|'PASSWD:'|'NOEXEC:'|'EXEC:'|'SETENV:'|'NOSETENV:')* Cmnd
- HTTP Proxy — When using a HTTP proxy defined through the variable
http_proxy
, you have to add/change the following lines to /etc/sudoers:
Defaults env_reset, env_keep=http_proxy
File Systems & Backup
See page Linux Disk Management.
sysfs and configfs
Reference:
The virtual filesystem sysfs contains device and driver information as well as various kernel data. sysfs provides an interface between kernelspace and userspace.
- /sys/block/
- This directory contains shortcuts to the sysfs files of the selected block device.
- /sys/bus/
- This directory contains the sysfs files and data for the different buses on the system.
- /sys/class/
- This directory contains folders named by device type like "printers", "mem", "leds", "input", etc. The subdirectories then contain shortcuts to the sysfs files pertaining to the selected device. This is useful for when a user is looking for a particular device. They can then go through this route.
- /sys/dev/
- Inside, there are two folders - "block" and "char" which direct users to the block and character devices respectively.
- /sys/devices/
- Most of the symbolic/soft links (shortcuts) in the sysfs system link to devices and files here.
- /sys/firmware/
- Files containing information and settings for the system's firmware resides here.
- /sys/fs/
- Information and settings pertaining to each individual mounted filesystem is placed here by filesystem type. This means there is a folder titled "ext4" which holds the folders for each device/partition with ext4.
- /sys/hypervisor/
- Data pertaining to the hypervisor is stored here.
- /sys/kernel/
- Settings, information, and security policies reside in this folder.
- /sys/module/
- All of the loaded modules can be seen here. Each folder contains information and settings for that particular module.
- /sys/power/
- This directory contains files with information on the power state, the number of times the system hibernated/slept, etc.
configfs is also a virtual filesystem on memory. configfs is used to change, make, and delete kernel objects and data. configfs is a kernel manager in the form of a filesystem.
To mount configfs:
sudo mount -t configfs none /sys/kernel/config
Settings in configfs are usually set as (with root privilege):
echo "STRING" > /config/FILE/PATH
System
- 20 Linux System Monitoring Tools Every SysAdmin Should Know
- 20 Command Line Tools to Monitor Linux Performance
top
,vmstat
,lsof
,tcpdump
,netstat
,htop
,iotop
,iostat
,iptraf
,psacct
oracct
, Monit,nethogs
,iftop
, Monitorix, Arpwatch, Suricata, VnStat PHP, NMon, Collectl.
Some more tools:
System monitoring
- (CLI)
ps
— displays the processes
- (CLI)
free
— Display amount of free and used memory in the system
- (CLI)
vmstat
— system activity, hardware and system information
- (CLI)
strace
— trace system calls and signals
- (CLI)
ltrace
— trace calls to dynamic libraries
- /proc file system — various kernel statistics
cat /proc/stat
cat /proc/$PID/stat
cat /proc/cpuinfo
cat /proc/meminfo
cat /proc/zoneinfo
cat /proc/mounts
- (CLI)
mpstat
— multiprocessor usage
- Part of sysstat package.
- (CLI)
pmap
— report memory map of a process
- (CLI)
smem
a more clever tool to monitor memory usage
- Can report RSS, but also PSS (proportional set size), a more meaninful representation of memory used [2]
- Nice!
- (TUI)
top
— process activity command
- (TUI)
atop
— AT Computing's System & Process Monitor
- BEST! An even better top program, with disk io stats, busy rate... Very easy to see what's going on. Use h for help, t to trigger update, 1 to switch to value/sec.
- (TUI)
htop
— an interactive process viewer for Linux (SourceForge)
- Nice! An improved top program.
- (CLI)
sar
— collect and report system activity
- Part of sysstat package.
- sysstat suite, containing
sar
,iostat
,tapestat
,mpstat
,pidstat
,sadf
,cifsiostat
.
- Nice! With data collection...
- (GUI) KDE System Guard - Real-time Systems Reporting and Graphing
- (GUI) Gnome System Monitor - Real-time Systems Reporting and Graphing
- (GUI)
Conky
— Lightweight system monitoring for X
- (Web-UI) Nagios Server And Network Monitoring
- (Web-UI) Cacti - Web-based Monitoring Tool
- (Web-UI)
munin
- BPF-based: 0x.tools, otel-profiling-agent
- More info on BPF: Linux Extended BPF (eBPF) Tracing Tools. See also HN comments
Disk / file monitoring
- (CLI)
iostat
— average cpu load, disk activity
- Part of sysstat package.
lsof
— list open files
lsof -f | grep bash
lsof -p $$ # view all files opened by current shell
Network monitoring
- (CLI)
netstat
— Print network connections, routing tables, interface statistics, masquerade connections, and multicast membershipsand
- (CLI)
ss
— another utility to investigate sockets
ss -tupan # Roughly equivalent to netstat -lpn
- (CLI)
iptraf
— real-time network statistics
- (CLI)
tcpdump
— detailed network traffic analysis
ifstat
— shows network traffic by interface in a vmstat/iostat-like manner
- (CLI)
nethogs
— Nethogs monitors traffic going to/from a machine, per process.
- Nice!
- (TUI)
iotop
— simple top-like I/O monitor
- (TUI)
bmon
— bandwidth monitor and rate estimator
- Nice! A kind of console mode darkstat.
- (TUI)
iftop
— display bandwidth usage on an interface
- shows network traffic by service and host
- (GUI) darkstat— Captures network traffic, calculates statistics about usage, and serves reports over HTTP.
- Breaks down traffic by host, protocol, etc. Geared towards analysing traffic gathered over a longer period, rather than `live' viewing.
- Nice!
- (Web-UI) ntopng — High-speed web-based traffic analysis and flow collection tool
- See ntopng.
- (Web-UI)
ntop
Network tools
- ettercap is a network sniffer/interceptor/logger for ethernet
Other tools
w
— find out who is logged on and what they are doinguptime
— tell how long the system has been runninguptimed
anduprecords
— daemon to record uptime records
uprecords -a -B
# # Uptime | System Boot up
# ----------------------------+---------------------------------------------------
# -> 1 0 days, 00:35:30 | Linux 4.19.0-2-amd64 Wed Mar 27 09:12:36 2019
# 2 1 day , 02:42:32 | Linux 4.19.0-2-amd64 Tue Mar 26 06:29:24 2019
# 3 9 days, 09:16:49 | Linux 4.19.0-2-amd64 Sat Mar 16 21:12:05 2019
Monitor a process activity
- strace is the tool of choice to monitor a process activity.
- Detect library usage (to debug missing library for instance)
- Detect all IO activity
- And more...
strace myprogram
strace -p $PID
strace -f myprogram # Follow child processes
- lsof (list open files) to view files opened by a given process, etc:
lsof -f | grep rsync
lsof -p $$
- ldd and readelf to list libraries
ldd /bin/ls # List libraries loaded by given executable
readelf -a /bin/ls # List libraries, and much more
- Monitor access to log file. For instance with:
ls -lS /var/log/*log | head
- See
apropos process
for even more tools.
apropos process
# ...
# fuser (1) - identify processes using files or sockets
# ...
- st (space-time) that monitors memory usage over time, and number of forks/threads.
Stop / continue a process
pkill -TSTP process # STOP gracefully
pkill -STOP process # STOP forcefully
pkill -CONT process # Resume (CONT) a stopped process
IO usage tips
To view IO usage of a specific process, check /proc/self/io, /proc/...pid.../io:
cat /proc/1234/io # View usage of process pid 1234
cat /proc/$(pgrep '^dd$')/io # View usage of process(es) named 'dd'
cat /proc/self/io # View udage of current process
# rchar: 2012
# wchar: 0
# syscr: 7
# syscw: 0
# read_bytes: 36864
# write_bytes: 0
# cancelled_write_bytes: 0
Memory usage tips
Free buffers and cache
- To free buffers and cache [3]
free && sync && echo 3 > /proc/sys/vm/drop_caches && free
Get peak memory usage of a process
From StackExchange [4]:
- Use cgmemtime, measures the high-water RSS+CACHE memory usage of a process and its descendant processes.
- Use kernel cgroup to measure memory usage of process and descendants:
sudo ./cgmemtime --setup -g myusergroup --perm 775
# This created /sys/fs/cgroup/memory/cgmemtime
./cgmemtime ./testa x 10 20 30
- Use tstime.
tstime date
- Use memusg, a script that does polling.
- However polling is a poor solution to measure peak statistics.
- Use standard
/usr/bin/time
:
/usr/bin/time -v sort myfile
Manage Out-of-Memory Killer
When the system is getting out-of-memory (OOM), the kernel triggers the OOM killer. The OOM killer computes the oom_score
of every process, and kill the process with highest score. This score can be adjusted through oom_adj
(ranging from -16 to +15) or oom_score_adj
(rangin from -1000 to +1000)
To see the oom_score
of every process [5]:
# Print directly from /proc
cat /proc/*/oom_score
# A nicer list with pid / oom_score / process name
while read -r pid comm; do
printf '%d\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$comm"
done < <(ps -e -o pid= -o comm=)
Likewise the oom_adj
oom_score_adj
are found in /proc:
cat /proc/*/oom_adj
cat /proc/*/oom_score_adj
To change the adjust value [6]:
# Change oom_score_adj of current process
echo 1000 > /proc/$$/oom_score_adj # OR
echo 1000 > /proc/self/oom_score_adj
# Change score on process startup
choom -n 1000 COMMAND [OPTION...]
Note that the oom_score_adj
are inherited by forked processes. Also, setting negative values requires root access.
Send commands via D-BUS
See D-Bus.
Power management
See Power management.
BIOS update
Using fwupdmgr
(for system BIOS, Thunderbolt docking station...).
- Tested successfully on HP EliteBook G4 + Thunderbolt docking station.
# Refresh DB
sudo fwupdmgr refresh
# Get list of available update
sudo fwupdmgr get-upgrades
# Update all devices
sudo fwupdmgr update
Hardware
Links
- Speaks about
lspci
but also GUI tool hardinfo.
Info / troubleshooting commands
Interesting commands:
- lshw, list hardware
lshw -C network
lshw -C display # See video controller
lshw -C display | grep driver # ... see driver in use
- lspci, listing all PCI devices
lspci
lspci | grep -i wireless # Write down sloce id of device
lspvi | grep -i vga # See video controller
sudo lspci -vv -nn -s 0c: # Slot id 0c:...
- lsmem, listing the ranges of available memory with their online status
- lscpu, displaying information about the CPU architecture
- inxi, Command line system information script for console and IRC
inxi -Gx # as current X desktop user, no sudo
- lsmod, Show the status of modules in the Linux Kernel
lsmod | sort
- modprobe, add and remove modules from the Linux kernel
- modinfo, show information about a kernel module (incl. available parameters)
modinfo iwlagn
- dmidecode, DMI (SMBIOS) table decoder
- To get BIOS version information:
sudo dmidecode | grep "BIOS Information" -A3
# BIOS Information
# Vendor: Hewlett-Packard
# Version: L71 Ver. 01.30
# Release Date: 12/09/2014
- uname, print system information
uname -rm
- lsusb, list USB devices
lsusb
# Bus 008 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
# Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
# Bus 006 Device 008: ID 056e:0056 Elecom Co., Ltd
# Bus 006 Device 006: ID 1131:1004 Integrated System Solution Corp. Bluetooth Device
lsusb -t
# /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ohci_hcd/8p, 12M
# |__ Port 1: Dev 2, If 0, Class=HID, Driver=usbhid, 12M
# /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci_hcd/8p, 480M
- USB devices are located at /dev/bus/usb, according to their bus and device id (for instance /dev/bus/usb/001/002 for bus 001, dev 002)
- usb-ctrl, control USB power (compiled from hub-ctrl.c, see USB)
usb-ctrl -v # Get list of devices and status
usb-ctrl -b 1 -d 1 -P 1 # Shut down port 1 on bus 1, dev 1
- usb-devices, list USB devices
usb-devices
# T: Bus=01 Lev=02 Prnt=02 Port=03 Cnt=02 Dev#= 4 Spd=12 MxCh= 0
# D: Ver= 2.00 Cls=ff(vend.) Sub=01 Prot=01 MxPS=64 #Cfgs= 1
# P: Vendor=413c ProdID=8197 Rev=01.12
# S: Manufacturer=Dell Computer Corp
# S: Product=DW380 Bluetooth Module
# S: SerialNumber=2016D895569F
# C: #Ifs= 4 Cfg#= 1 Atr=e0 MxPwr=0mA
# I: If#= 0 Alt= 0 #EPs= 3 Cls=ff(vend.) Sub=01 Prot=01 Driver=btusb
# I: If#= 1 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=01 Prot=01 Driver=btusb
# I: If#= 2 Alt= 0 #EPs= 2 Cls=ff(vend.) Sub=ff Prot=ff Driver=(none)
# I: If#= 3 Alt= 0 #EPs= 0 Cls=fe(app. ) Sub=01 Prot=01 Driver=(none)
- Debug information
dmesg | egrep -i "wlan|iwl"
- To test availability of OpenGL:
glxinfo | grep -i direct
# The result should be:
# direct rendering: Yes
- Input devices in /proc (some not listed by commands above — e.g. synaptic touchpad):
cat /proc/bus/input/devices # List input devices
- udevadm — Query udev database
udevadm info -q all -n /dev/bus/usb/002/001 # Can query using the device name
udevadm info -q all -p /sys/bus/usb/devices/usb2 # Or using udev device paths (or symlink)
udevadm info -a -n /dev/bus/usb/002/040 # Walk up the chain of parent device, and for each, show attributes
- View DHCP lease info on client:
cat /var/lib/dhcp/dhclient.leases
WiFi
See Wifi.
Firmware
Install package firmware-linux or firmware-linux-nonfree to update hardware with latest firmware
sudo apt install firmware-linux-nonfree
# sudo apt install firmware-linux
Sometimes we get messages like the following when running apt
(actually generated by update-initramfs
)
update-initramfs: Generating /boot/initrd.img-5.17.0-1-amd64 W: Possible missing firmware /lib/firmware/i915/adlp_dmc_ver2_14.bin for module i915
We can find the latest available firmware on the kernel git linux-firmware.git.
To fix the message, copy the firmware to /lib/firmware (see also note below), and reconfigure the package
sudo cp *.bin /lib/firmware/i915 # for i915 firmware
sudo dpkg-reconfigure linux-image-5.17.0-1-amd64 # Trigger initramfs again
- Debian
(From Debian Wiki) firmware is sourced from the following places (see udev's /lib/udev/hotplug.functions
and /lib/udev/firmware.agent
):
- /lib/firmware/$(uname -r) - Firmware provided by a package, specific for a kernel.
- /lib/firmware/ - Firmware provided by a package, valid for all kernels.
- /usr/local/lib/firmware - Location for manually installed firmware.
- NOTE: It seems this directory is ignored, we still get the msg about missing firmware, and the message disappear if we copy it in /lib/firmware/i915
- /usr/lib/hotplug/firmware - Firmware provided by a package, valid for all kernels
Wireless firmware
Locate your wireless card:
lspci | grep -i network
# 0c:00.0 Network controller: Intel Corporation PRO/Wireless 5300 AGN [Shiloh] Network Connection
# =======
lspci -s 0c: -v
#0c:00.0 Network controller: Intel Corporation PRO/Wireless 5300 AGN [Shiloh] Network Connection
#...
# Kernel modules: iwlagn
# ======
To get driver version:
sudo lshw -C network
# configuration: broadcast=yes driver=iwlagn driverversion=2.6.32-32-generic-pae firmware=8.24.2.12 latency=0 link=yes multicast=yes wireless=IEEE 802.11abgn
# ==================
Print module information, the loaded firmware is located at the very beginning:
modinfo iwlagn
# filename: /lib/modules/2.6.32-32-generic-pae/updates/compat-wireless-2.6.37/iwlagn.ko
# ====================== ----------------> interesting
# description: Intelsl(R) Wireless WiFi Link AGN driver for Linux
# ...
# firmware: iwlwifi-4965-2.ucode \
# firmware: iwlwifi-5150-2.ucode |
# firmware: iwlwifi-5000-2.ucode |
# firmware: iwlwifi-130-5.ucode |
# firmware: iwlwifi-6000g2b-5.ucode |
# firmware: iwlwifi-6000g2a-5.ucode |___ The firmwares used by this driver
# firmware: iwlwifi-6050-5.ucode |
# firmware: iwlwifi-6000-4.ucode |
# firmware: iwlwifi-100-5.ucode |
# firmware: iwlwifi-1000-3.ucode /
For INTEL cards:
- The original drivers (microcode) are available on IntelLinuxWireless.org (downloads).
These are the iwlwifi*.ucode file. But as of kernel 2.6.24, these cannot be used as is. - Instead use the code from the compat-wireless project, in particular the one from linux 2.6.all stable compat-wireless releases.
See nxl65002ux for example on how to upgrade the wireless driver
Other links:
- Intel WiFi Link 5100/5300 WLAN controller on ThinkWiki.org, with detailed instructions on how to install iwlagn + ucode firmware — not so clear after all...
System information
- lshw is available by default,
- or use sysinfo (sudo apt-get install sysinfo),
- or use hardinfo (sudo apt-get install hardinfo),
To dump BIOS data:
sudo dmidecode
sudo dmidecode -t system
sudo dmidecode -t system | grep Serial
, this returns the serial number label on HP computer for instance.
udev & devfs
Reference: [7]
This chapter is about the devices in /dev. Since kernel 2.6, the content of this directory is generated by udev rules.
These rules are located at:
- /lib/udev/rules.d
- /etc/udev/rules.d (these can be customized)
Use udevadm to get information on a given device:
udevadm info -q path -n /dev/sda2 # To get the path to the device /dev/sda2
udevadm info -q -n /dev/sda2 # To get all the attributes of device /dev/sda2
udevadm info -a -p $(udevadm info -q path -n /dev/sda2) # Same as above
udevadm test $(udevadm info -q path -n /dev/sda2) 2>&1 | grep OWNER # Test the effect of a new rule on device /dev/sda2
USB Serial (FTDI, Prolific)
The usbserial module (usbserial.ko) provides a generic serial interface (aka virtual COM port in windows) for USB devices. They are basically two (low cost) chip providers on the market: FTDI and Prolific. For FTDI chips, drivers are included in kernel since 2.6.31. Module is called ftdi_sio, and exposes FTDI devices as generic usbserial device.
When connecting the device, the following can be seen in /var/log/messages
dmesg
usb 7-1: new full speed USB device using uhci_hcd and address 18
usb 7-1: configuration #1 chosen from 1 choice
ftdi_sio 7-1:1.0: FTDI USB Serial Device converter detected
usb 7-1: Detected FT232RL
usb 7-1: Number of endpoints 2
usb 7-1: Endpoint 1 MaxPacketSize 64
usb 7-1: Endpoint 2 MaxPacketSize 64
usb 7-1: Setting MaxPacketSize 64
usb 7-1: FTDI USB Serial Device converter now attached to ttyUSB0
usb 7-1: USB disconnect, address 18
Now, the device is available as /dev/ttyUSB0
- Problems, Issues, Workaround
- If you see a disconnect message, and no /dev/ttyUSB0 showing up:
ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0
ftdi_sio 7-1:1.0: device disconnected
- The most probable cause is that you have package brltty installed. This package automatically identifies device with an FTDI chip as a braille device. The work-around is simply to uninstall the package.
- VirtualBox might also interfere with the usb device. User can give a list of USB devices that a virtual machine should immediately connect to when the device is plugged in. If the FTDI device is listed, VirtualBox will grab the device and it will not be available to the host. Simply unlist the device to get it back available on the host.
- Other potential conflicts around libusb, usbfs, libftdi
- Unsufficient access rights (typically user must be member of group dialout)
ls -l /dev/ttyUSB*
# crw-rw---- 1 root dialout 188, 0 2011-09-22 17:26 /dev/ttyUSB0
sudo gpasswd -a $USER dialout
id
# uid=6659(beq06659) gid=6659(beq06659) groups=4(adm),20(dialout),...
- Links
- Arduino board install guide
- USB-to-serial adapter for the Armadeus board
- libFTDI, a linux library to talk to FTDI chips (and using libusb).
- FTDI
- The Linux USB sub-system (apparently pre-udev)
- Advanced Kung-fu (Create a symlink with udev when attaching a device)
find /sys -name ttyUSB*
#Look up the attributes of the directory that should (hopefully) be listed (following is guestimate).
udevinfo -a -p /sys/class/tty/ttyUSB0
# pick a unique attribute to copy.
sudo gedit /etc/udev/rules.d/60-symlinks.rules
# Add lines:
# # Create /dev/bstamp symlink for FTDI Device
# KERNEL=="ttyUSB*", ATTRS{product}=="FT232RL", \
# SYMLINK+="bstamp"
#
# ATTRS{attr} will be the information taken from the undevinfo command.
sudo /etc/init.d/udev restart
- USB Serial on cygwin
Some interesting links:
- This explains that after setting up manually the COM: port number in windows (say COM16), one can access that port in cygwin by using /dev/ttyS15 or /dev/com16. However there might be a limitation that cygwin only allow up to 16 serial ports (from /dev/com1 up to /dev/com16), but maybe this limitation is not there anymore.
- Some answer in the post saying that cygwin support more than 16 ports (it mentions /dev/ttyS26). However it mentions also another issue that the input blocking until a CR is received.
- Some post from Corinna Vinschen, warning not to use windows names \\.\COMx or COMx in cygwin, or will not get any POSIX Serial I/O support from Cygwin.
NVidia
Check nvidia website for detailed information on nvidia drivers for linux:
Some tips:
- Use xdpyinfo to show the current extensions. For OpenGL, it should show extension "glx" and "nv-glx".
- Check dmesg for error messages related to nvidia
- Force reload of module with
modprobe nvidia
- Prevent Nouveau from being loaded. Create a file /etc/modprobe.d/disable-nouveau.conf:
blacklist nouveau
options nouveau modeset=0
- Note that this will not prevent the X server to load Nouveau. If loaded, this can be unloaded with
modprobe -r nouveau
, as long as Nouveau has been prevented from doing a kernel modeset.
My /etc/X11/xorg.conf on Maverick:
Section "Screen"
Identifier "Default Screen"
DefaultDepth 24
EndSection
Section "Module"
Load "glx"
EndSection
Section "Device"
Identifier "Default Device"
Driver "nvidia"
Option "NoLogo" "True"
EndSection
BlueTooth
- Install BlueTooth Manager to solve bluetooth connection issue (package blueman)
Udisks
The [udisks daemon serves as an interface to system block devices. It is responsible to mount automatically inserted DVDs.
- Troubleshooting - Fix broken permissions on UDF
- Some DVD recorder do not set correctly the directory permissions on DVDs (UDF file system) (missing 'x' flags)
- There is a fix in udisk that force directory permissions to 0500, but only if DVD is read-only (bug 635499, see fix [10])
- The fix does not work for DVD-RW that are not finalized yet. These discs are then not readable on Linux.
- There is no way apparently to override the default mount options in udisks [11], [12]). So our only hope is to patch the udisks package directly.
- The following is a patch on udisk=1.0.4-5ubuntu2.1 to force dmode=0500 for all optical discs with UDF file systems (to rebuild a package, see this page):
diff --git a/src/device.c b/src/device.c
index a7f8880..3174628 100644
--- a/src/device.c
+++ b/src/device.c
@@ -6204,10 +6204,8 @@ struct Job
/* dynamic default options */
/* some broken DVDs come with 0400 directory permissions, making them
- * unreadable; overwrite readonly UDF media with a 0500 dmode. */
- if (g_strcmp0 (device->priv->id_type, "udf") == 0 && device->priv->device_is_optical_disc &&
- device->priv->drive_media != NULL &&
- strstr(device->priv->drive_media, "_rw") == NULL && strstr(device->priv->drive_media, "_ram") == NULL)
+ * unreadable; overwrite all UDF media with a 0500 dmode. */
+ if (g_strcmp0 (device->priv->id_type, "udf") == 0 && device->priv->device_is_optical_disc)
{
g_ptr_array_add (options, g_strdup("dmode=0500"));
}
- Install the fixed package, and check that mount has the new dmode=0500 flag:
/dev/sr0 on /media/DVD VR type udf (ro,nosuid,nodev,uid=6659,gid=6659,iocharset=utf8,umask=0077,dmode=0500,uhelper=udisks)
- Alternate patch. This force flag dmode=0500 for all UDF file system, similar to iso9660:
diff --git a/src/device.c b/src/device.c
index 2d7621b..a7f8880 100644
--- a/src/device.c
+++ b/src/device.c
@@ -5926,7 +5926,7 @@ struct Job
/* ---------------------- udf -------------------- */
-static const char *udf_defaults[] = { "uid=", "gid=", "iocharset=utf8", "umask=0077", "dmode=0500", NULL };
+static const char *udf_defaults[] = { "uid=", "gid=", "iocharset=utf8", "umask=0077", NULL };
static const char *udf_allow[] = { "iocharset=", "umask=", "mode=", "dmode=", NULL };
static const char *udf_allow_uid_self[] = { "uid=", NULL };
static const char *udf_allow_gid_self[] = { "gid=", NULL };
@@ -6203,6 +6203,15 @@ struct Job
/* dynamic default options */
+ /* some broken DVDs come with 0400 directory permissions, making them
+ * unreadable; overwrite readonly UDF media with a 0500 dmode. */
+ if (g_strcmp0 (device->priv->id_type, "udf") == 0 && device->priv->device_is_optical_disc &&
+ device->priv->drive_media != NULL &&
+ strstr(device->priv->drive_media, "_rw") == NULL && strstr(device->priv->drive_media, "_ram") == NULL)
+ {
+ g_ptr_array_add (options, g_strdup("dmode=0500"));
+ }
+
/* user supplied options */
for (n = 0; given_options[n] != NULL; n++)
{
For udisk2:
diff --git a/src/udiskslinuxfilesystem.c b/src/udiskslinuxfilesystem.c
-static const gchar *udf_defaults[] = { "uid=", "gid=", "iocharset=utf8", "umask=0077", NULL };
-static const gchar *udf_allow[] = { "iocharset=", "umask=", NULL };
+static const gchar *udf_defaults[] = { "uid=", "gid=", "iocharset=utf8", "umask=0077", "dmode=0500", NULL };
+static const gchar *udf_allow[] = { "iocharset=", "umask=", "dmode=", NULL };
Software
Packages
See page Package Management.
Libraries
See the Library HOWTO.
Static Libraries
See Library HOWTO - Static Libraries
See Library HOWTO - Shared Libraries
Path conventions according to the info:standards#Directory_Variables GNU Standards (used by developers):
- /usr/local/lib: for all libraries when distributing source code (executables go to /usr/local/bin).
Path conventions according to the Filesystem Hierarchy Standard) (used by distributors through package management)
- /usr/lib: for most libraries (executables go to /usr/bin, executables that users should not call directly go to /usr/libexec/).
- /lib: for libraries needed at boot time.
- /usr/local/lib: for libraries that are not part of the system (/usr/local/bin for executables, and /usr/local/libexec for library executable)
- soname — real name — linker name
- /usr/lib/libreadline.so.3 is a fully-qualified soname (symlinked to realname below by ldconfig)
- /usr/lib/libreadline.so.3.0 is the realname
- /usr/lib/libreadline.so is the linker name (symlinked to soname /usr/lib/libreadline.so.3)
- Environment variables
- LD_LIBRARY_PATH temporarily overrides the usual library path for a given executable (should only be used for debugging)
- LD_DEBUG triggers debugging in C loader (e.g.
LD_DEBUG=files /bin/ls
)
- Utilities
ldconfig -n directory_with_shared_libraries #Creates soname links to realname when installing new libraries
ldd /bin/ls #List shared libraries needed by a given executable
- List symbols exported by libraries
From SO:
nm -g yourLib.so # Standard tool
nm -gC yourLib.so # ... C++ lib, demangle the symbols
objdump -TC libz.so # For library in ELF format - -C to demangle
readelf -Ws libz.so # ... Another solution for ELF
readelf -a /some/exec
Dynamically Loaded (DL) Libraries
See Library HOWTO - Dynamically Loaded Libraries.
Managing Alternatives
For instance, to define the default cursor-theme, use update-alternatives
:
sudo update-alternatives --config x-cursor-theme
Troubleshooting
Use LD_PRELOAD to fix symbol '...': can't resolve symbol
cat bup_intl_issue.py
# import posix1e; # From package pylibacl
python bup_intl_issue.py
# /opt/bin/python2.7: symbol 'libintl_gettext': can't resolve symbol
# Traceback (most recent call last):
# File "bup_intl_issue.py", line 1, in <module>
# import posix1e;
# ImportError: unknown dlopen() error
We can fix the issue using the LD_PRELOAD
trick though:
LD_PRELOAD=/opt/lib/libintl.so python2.7 bup_intl_issue.py
No such file or directory
when executing a program
Bash returns an error No such file or directory
even though the executable is there:
./lmgrd
# bash ./lmgrd: No such file or directory
ls -l ./lmgrd
# -rwxr-xr-x 1 superman superman 1.5 Nov 21 2016 ./lmgrd
The problem is that the executable requires a library that is not present (likely the interpreter):
readelf -a ./lmgrd | grep -i interpreter # Could also do 'ldd ./lmgrd'
# [Requesting program interpreter: ./lib64/ld-lsb-x86-64.so.3]
cd /lib64
sudo ln -sf /lib/x86_64-linux-gnu/ld-2.19.so ld-lsb-x86-64.so.3
Troubleshooting process using libusb
strace
is the tool of choice.
strace someprocess &> strace.txt
- We look for the application error messages in strace.txt.
- Look at system calls that happened before that error messages. Is the errror due to bad permission, or device busy?
- Bad permission, check device permission (usually /dev/bus/usb). Probably the device needs a udev rule to setup the permission.
- Device busy, check for which module is grapping the device. Maybe this device must be black-listed in /etc/modprobe.d.
- Look which libraries are loaded (calls to
open
). Are all libraries found?
It might be necessary to follow child processes. Use stace -f
for this.
Another useful tool is strace
(to trace calls to dynamically-linked libraries).
Network
See also Linux networking and Wifi.
Commands
ZeroConfig
ZeroConfig refers to all utilities that help setting up network without any additional configuration. Somes links:
- wikipedia
- ZeroConf on debian.org.
- zeroconf.org website (open standard issued by Apple).
To install on Linux (Ubuntu):
sudo apt-get install libnss-mdns avahi-daemon mdns-scan
- Address resolution
- To be completed.
- Name resolution
- By default, any given host can be accessed via the name
hostname.local
without need of a local DNS server.
ping myserver.local
- Name resolution relies on mDNS (multicast DNS) protocol. mDNS client makes a request to a well-known multicast address (224.0.0.251 for IPv4 and ff02::fb for IPv6 link-local addressing). On Linux, avahi package implements the Apple Zeroconf specification.
- To list all names that are broadcasted:
mdns-scan
- Service discovery
- To be completed?
Network Manager - Search Path
See NetworkManager Ubuntu documentation for how to add a static local domain to resolv.conf search path.
Basically:
- In the NM applet, changed the network from DHCP (auto) to DHCP (address only)
- Edit the network configuration file in /etc/NetworkManager/system-connections to appear as follows:
- Select the network in the wired network
[ipv4] method=auto dns-search=domain1.com;domain2.org;domain3.edu; ignore-auto-routes=false ignore-auto-dns=false # !!! Set this line back to FALSE !!!
Import Windows Settings for Enterprise Wireless Network (Dynamic WEP or WPA & WPA2 Enterprise, TLS)
This chapter explains how to import the network configuration settings from Windows for an enterprise wireless network using Dynamic WEP (802.1x), with TLS authentication.
- In Windows, export the client Authentication certificate and private key from Windows Certificate Store:
- In Control Panel → Internet Options → Content tab, click Certificates, or
alternatively, type Win-R, certmgr.msc. - In the Personal tab, select the certificate used for client authentication, and click Export.
- In the new window, click Next, then select Yes, export the private key and click Next.
- If the option export the private key is greyed out, see Windows Administration.
- Select format Personal information interchange - PKCS #12 (.PFX), and select Include all the certificates in the certificate path if possible and Enable strong protection.
- Select a password, and save the file (say mywindowscert.pfx).
- In Ubuntu, split the exported certificate in the components CA / Cert / Private key.
- See SSL.
- Now create a new wireless network connection in Ubuntu:
- Security: Dynamic WEP (802.1x) or WPA & WPA2 Enterprise
- Authentication: TLS
- Identity: the account name (this is not necessarily the same as the name whom the certificate was issued to)
- User Certificate: mycert.crt.pem
- CA certificate: mycert.ca.pem
- Private key: mycert.key.p12
- Private key password: as required
More information:
- https://wiki.freeradius.org/protocol/EAP-PEAP#types_peap-eap-tls
- https://en.wikipedia.org/wiki/Protected_Extensible_Authentication_Protocol
Firewall
To troubleshoot firewall connection issues:
- See firewall log (ufw.log for UFW)
- Use
netstat
:
#For instance, troubleshooting Samba server firewall issues:
service smb stop
netstat -ln > netstat-ln-smb.before
service smb start
netstat -ln > netstat-ln-smb.after
diff -u netstat-ln-smb.*
Network Time Protocol (ntp)
See ntp.
Chroot
See Chroot page.
TTY
Stuff to read some day:
- The TTY Demystified
- Stack Overflow - How does Ctrl-C terminate a child process
- Why is SIGINT not propagated to child process when sent to its parent process
Printing
CUPS
- Current print server on Ubuntu is CUPS
- Web Interface — CUPS server is accessed via at http://localhost:631 (see CUPS-Print Server)
- Reset CUPS printer
- Go to http://localhost:631 to reset CUPS printer
- More troubleshooting
- See [13]
- Setup backend error handler
- See [14], [15]
- beh is already installed on Ubuntu 12.04+.
- Another solution is to edit the policy ([16]).
- Default policy is retry-job, but I still get error message file is rejected. I tried then abort-job.
- Command line interface
# Get printers status
lpstat -p
# printer Canon_MG5300_series is idle. enabled since Wed 25 Dec 2019 06:52:57 PM CET
# printer DESKJET-940C is idle. enabled since Wed 25 Dec 2019 06:50:32 PM CET
# printer TS8100 is idle. enabled since Tue 13 Aug 2019 04:22:10 PM CEST
# printer Canon_MG5300_series disabled since Wed 25 Dec 2019 06:52:37 PM CET -
# Paused
# printer DESKJET-940C is idle. enabled since Wed 25 Dec 2019 06:50:32 PM CET
# printer TS8100 is idle. enabled since Tue 13 Aug 2019 04:22:10 PM CEST
# printer Canon_MG5300_series is idle. enabled since Wed 25 Dec 2019 06:13:46 PM CET
# printer DESKJET-940C now printing DESKJET-940C-288. enabled since Wed 25 Dec 2019 06:47:01 PM CET
# Processing page 2...
# printer TS8100 is idle. enabled since Tue 13 Aug 2019 04:22:10 PM CEST
# Enable paused printer
cupsenable DESKJET-940C
Rescue
Some tips to rescue a broken linux installation.
Using GRUB
See Grub#Rescue on how to fix a broken GRUB installation or on how to use GRUB to fix a broken linux installation.
Kernel line
To boot a minimal bash shell, edit the kernel line as follows:
- Change ro → rw to allow read-write access to file system
- Add init=/bin/bash to run Bash shell
After that, one can uses eg. nano to edit text configuration files.
To get boot messages:
- Remove quiet splash
- Add --verbose
Alt-SysRq-REISUB / Alt-PrtScr-REISUB
Using the Magic SysRq key, one can usually reboot its system nicely (better than holding the power button for 5sec).
Press and hold Alt-SysRq, then press the following keys in sequence, waiting 1 second between each press: R, E, I,S, U, B.
On laptop, you often have to press and hold Alt, then Fn-SysRq, then release Fn while holding Alt-SysRq, and finally R E I S U B.
On keyboard without SysRq, the combination Alt-PrtScr also works.
unRaw (take control of keyboard back from X), tErminate (send SIGTERM to all processes, allowing them to terminate gracefully), kIll (send SIGKILL to all processes, forcing them to terminate immediately), Sync (flush data to disk), Unmount (remount all filesystems read-only), reBoot.
A mnemonic "Reboot Even If System Utterly Broken", or BUSIER when read backwards
/proc/sysrq-trigger
SysRq can also be triggered by writting to /proc/sysrq-trigger [17]:
echo u | /proc/sysrq-trigger # Remount all fs read-only
Troubleshooting
References:
- KernelDebuggingTricks (wiki.ubuntu.com)
Serial Console
References:
- SerialConsoleHowto (help.ubuntu.com)
- How to redirect the Linux guest kernel console to a serial port / file (www.virtualbox.org)
- [Setting up serial console on VirtualBox
Use the serial console to debug kernel freezes, hangs... that leaves no traces in the kernel logs.
To enable the serial console, you need to have a serial port (named /dev/ttyS0 on Linux, COM1 on windows), and have a process like minicom that connects to that port on the machine to debug.
To enable, edit /etc/default/grub, and add the line:
-GRUB_CMDLINE_LINUX_DEFAULT="splash quiet"
+GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS0 console=tty0 ignore_loglevel"
The first parameter will ensure that all kernel output is redirected to the serial console. The second parameter will ensure that the kernel output is still logged to the guest text console. The third parameter enforces the guest Linux kernel to print all kernel messages to the console.
Update grub and restart.
sudo update-grub
sudo shutdown -r now
- VirtualBox
On VirtualBox [18], one can easily configure a virtual serial port. In the settings, open the Serial Ports tab:
- Enable serial port COM1 (irq 4, io 0x3F8).
- Select mode Raw File, and select a file on the host file system.
Restart the guest, and verify that the port is running:
sudo stty -F /dev/ttyS0 -a
# speed 9600 baud; rows 0; columns 0; line = 0;
# intr = ^C; quit = ^; erase = ^?; kill = ^U; eof = ^D; eol = <undef>;
# eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
# werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
Another option is to use Host pipe, and use for instance minicom to attach to the named pipe [19], [20], [21].
Netconsole
References:
- Ubuntu Netconsole
- See Remote kernel logs with netconsole and rsyslog (2013)
- netconsole documentation
- dmesg(8) man page and Documentation/kernel-parameters.txt.
netconsole is similar to the serial console, except that messages are sent to a network socket instead of a serial link.
- Setup Receiver side
On the remote host, the simpler is to start a netcat process or equivalent to collect the log:
nc -l -p 6666 -u # ... OR ...
netcat -u -l -p 6666 # ... OR ...
nc -l -p 6666 -u > netconsole.txt &
A more stable solution is to use syslog-ng
(see Ubuntu wiki above) or rsyslog
(see Stapelberg's link above).
To setup rsyslog:
- First create the log folder and rsyslog config:
# Here we assume sender hostname is 'zavcxl0006' with IP 192.168.1.4
sudo mkdir -p /var/log/remote/zavcxl0006
cat >/etc/rsyslog.d/remote.conf <<EOT
\$ModLoad imudp
\$RuleSet remote
# For each IP address that you want to store logs from,
# add and modify the following two (!) lines:
if \$fromhost-ip=='192.168.1.4' then /var/log/remote/zavcxl0006/console.log
& stop
\$InputUDPServerBindRuleset remote
\$UDPServerRun 6666
\$RuleSet RSYSLOG_DefaultRuleset
EOT
- Create the logrotate config:
cat >/etc/logrotate.d/remote <<'EOT'
/var/log/remote/*/*.log
{
copytruncate
rotate 30
daily
missingok
dateext
notifempty
delaycompress
compress
maxage 31
postrotate
invoke-rc.d rsyslog reload > /dev/null
endscript
}
EOT
- Setup Sender side
To use it, first install it from debian packages:
sudo apt install netconsole
To enable netconsole you must change kernel option at boot time. Edit /etc/default/grub:
-GRUB_CMDLINE_LINUX_DEFAULT="splash quiet"
+GRUB_CMDLINE_LINUX_DEFAULT="debug ignore_loglevel"
Then update grub:
sudo update-grub
Or set it via dmesg
(see dmesg(8) man page and Documentation/kernel-parameters.txt for details)
netconsole can be started automatically with a static configuration. To start at boot time, update /etc/modules and create a config file in /etc/modprobe.d:
sudo sh -c 'echo netconsole >> /etc/modules'
sudo sh -c 'echo options netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82 > /etc/modprobe.d/netconsole.conf'
Or start it at command-line:
sudo modprobe netconsole netconsole=6666@192.168.1.102/eth0,6666@192.168.1.103/08:00:46:d4:1d:82
The syntax is:
netconsole="[src-port]@[src-ip]/[<dev>],[tgt-port]@<tgt-ip>/[tgt-macaddr][;...]"
# src-port source for UDP packets (defaults to 6665)
# src-ip source IP to use (interface address) - MANDATORY at boot time
# dev network interface (eth0)
# tgt-port port for logging agent (6666)
# tgt-ip IP address for logging agent
# tgt-macaddr ethernet MAC address for logging agent (broadcast)
Quotes are mandatory for multi-host configuration (semi-colon) but optional otherwise. Giving the src-ip is mandatory at boot time (when starting module in /etc/modules because the interface does not have an IP address yet).
netconsole can also be configured dynamically [22] using configfs [23]:
sudo mount -t configfs none /sys/kernel/config
cd /sys/kernel/config
sudo mkdir target_a
cd target_a
cat enabled # check if enabled is 1
echo 0 > enabled # disable the target (if required)
echo eth2 > dev_name # set local interface
echo 10.0.0.4 > remote_ip # update some parameter
echo cb:a9:87:65:43:21 > remote_mac # update more parameters
echo 1 > enabled # enable target again
Note netconsole module cannot be removed if configfs contains a dynamic target.
Alternatively, here a script to configure dynamically netconsole
module:
unset NETCON_TGTHOST
eval $(get-network.sh < ~/etc/network_definition)
case $LOCATION in
home)
# Require
# ipkg install netcat
# mkdir -p /shares/beq06659/netconsole
# nc -l -p 6666 -u >/shares/beq06659/netconsole/zavcxl0005-netconsole&
NETCON_TGTHOST=lacie-cloudbox
;;
stzav)
NETCON_TGTHOST=apple-pi.zav.st.com
;;
esac
echo "Resetting netconsole for location '$LOCATION'..."
sudo rmmod netconsole 2> /dev/null
if [ -n "$NETCON_TGTHOST" ]; then
NETCON_PORT=6666
# Adding -v -v to nc to avoid it to block sometimes. Go figure...
echo "[------------] Redocking on $(date +"%Y-%m-%d %H:%M:%S")..." | nc -v -v -u -p 6665 $NETCON_TGTHOST $NETCON_PORT
NETCON_TGTIP=$(getent hosts $NETCON_TGTHOST|awk '{print $1}')
ping -c 1 $NETCON_TGTIP >/dev/null
NETCON_TGTMAC=$(arp -n $NETCON_TGTIP|awk '/ether/{print $3}')
echo "... setting up netconsole for location '$LOCATION' ($NETCON_TGTHOST,$NETCON_TGTIP/$NETCON_TGTMAC)"
sudo modprobe netconsole netconsole=@/,$NETCON_PORT@$NETCON_TGTIP/$NETCON_TGTMAC
else
echo "... no target available for netconsole at location $LOCATION"
fi
On Debian, netconsole can be setup more easily using netconsole-setup
:
echo "Resetting netconsole for location '$LOCATION'..."
sudo rmmod netconsole 2> /dev/null
if [ -n "$NETCON_TGTHOST" ]; then
NETCON_PORT=6666
# Adding -v -v to nc to avoid it to block sometimes. Go figure...
echo "[------------] Redocking on $(date +"%Y-%m-%d %H:%M:%S")..." | nc -v -v -w 1 -u -p 6665 $NETCON_TGTHOST $NETCON_PORT
echo "... using target host '$NETCON_TGTHOST'."
sudo netconsole-setup 6666@$NETCON_TGTHOST
else
echo "... no target available for netconsole at location $LOCATION."
fi
- Troubleshooting
- If getting message
$ sudo modprobe netconsole netconsole=@/,6666@192.168.1.3/b8:27:eb:69:3e:df modprobe: ERROR: could not insert 'netconsole': No such device
- Try to add the name of the network interface to use (see [24]).
System Logs
Using systemd
To view kernel logs (with highlighting, grouping...):
sudo journalctl -b # Since last boot
# or
sudo journalctl -fe # Last event and follow
/var/log
Some external links:
- Quick HOWTO - Troubleshooting Linux with syslog
- System Log Viewer (package gnome-utils), is an handy tool to view log files under Gnome, with entries sorted by date, and new entries highlighted in bold.
- dmesg(8) man page and Documentation/kernel-parameters.txt.
syslog is an utility to log all system messages, from information messages to critical errors. Log files are stored in /var/log. On Ubuntu, the default logging system is rsyslog, with configuration files /etc/rsyslog.conf and in /etc/rsyslog.conf.d/.
Logs generated by rsyslog (see /etc/rsyslog.d/50-default.conf):
file | source | description |
---|---|---|
aptitude | ||
auth.log | rsyslog | Messages to facilities auth and authpriv |
boot | ||
boot.log | ||
btmp | ||
daemon.log | rsyslog | Messages to facility daemon |
debug | rsyslog | Messages with debug priority, but excluding facilities auth, mail and news |
dmesg | kernel | Boot time hardware detection and driver setup (i.e. kernel messages before syslog daemon is launched). This is *not* the same as dmesg output (see kern.log)!
|
dpkg.log | ||
faillog | ||
fontconfig.log | ||
jockey.log | ||
kern.log | rsyslog | Messages to facility kern (apparently dmesg will display the last 16392 octets of /var/log/kern.log since last boot [25])
|
lastlog | lastlog | last login of each user ([26]). It looks big, but it's a sparse file (du -h lastlog ) !!!
|
lpr.log | rsyslog | Messages to facility lpr |
mail.info | rsyslog | Messages to facility mail, priority ≥ info |
mail.err | rsyslog | Messages to facility mail, priority ≥ err |
mail.log | rsyslog | Messages to facility mail |
mail.warn | rsyslog | Messages to facility mail, priority ≥ warn |
messages | rsyslog | Messages with info,notice and warn priority, but excluding facilities auth, daemon, mail and news |
MountManager.log | ||
mysql.err | ||
mysql.log | ||
pm-powersave.log | ||
pm-suspend.log | ||
pycentral.log | ||
syslog.log | rsyslog | All messages except those in auth.log (i.e. facilities auth and authpriv) |
udev | ||
ufw.log | rsyslog | All messages from UFW firewall |
user.log | rsyslog | All messages targeting facility user |
vbox-install.log | ||
wtmp | ||
Xorg.0.log | ||
Xorg.failsafe.log |