Here is a list of some very useful command lines.

Shell

Redirections

  • <>&1 | tee file in/out printed both shell and file
  • 2>&1 | tee file all out printed both shell and file
  • 2> errors
  • >> append
  • > create (stdout only)
  • <> read / write
  • >& overwrite

Scripting and Text Extraction

  • Debugging shell script: sh -x script
  • Powerful scripting with awk: awk '{print $4}'
  • Cutting fields in line: cut -d';' -f2

Example: Summing a CSV file column

Given a CSV file with semi-columns as column separator, summing the 15th column:

awk -F ';' '{ s += $15 }  END { print s }' the_file.csv

Using comma as decimal separator:

LC_ALL=fr_FR awk -F ';' '{ s += $15 }  END { print s }' the_file.csv

macOS Only

  • Recover dead hard drive: dd bs=512 id=/dev/todo of=/some_dir/foo.dmg conv=noerrir,sync
  • (Old versions of macOS) Copy folder (keep HFS metadata): sudo ditto -rsrc "/from" "to"
  • Clipboard access: pbcopy, pbpaste
  • Testing network: networkQuality
  • Image manipulation and resizing: sips
  • Text, Word, HTML files conversion: textutil
  • Making screenshots: screencapture
  • Speech a text: say
  • Force iCloud synchronisation:
    • Download a local copy: brctl download /path/to/filename
    • Evict de local copy: brctl evict /path/to/filename
  • Homebrew, the package manager:
    • Updates: brew update
    • Upgrades: brew upgrade
    • Diagnostic: brew doctor
    • Start a service, such as PostgreSQL; brew services start postgres
  • Keep your Mac from sleeping: caffeinate
    • Keep your Mac awake for 1h: caffeinate -t 3600
    • Keep your Mac awake for executing a command: caffeinate -i your_long_scrtip.sh
    • Keep your Mac awake until the end of a process: caffeinate -w pid
  • Openning files or applications: open
    • Using an application for opening a file: open -a the_app the_file ; using -g for background
    • Open a Finder window within the current folder: open .
    • Open a Finder window within a gvien file: open -R the_file
  • Spotlight search: mdfind / mdls a_file for displaying Spotlight metadata
  • Process scheduling (silicon Macs):
    • Run in background (run on efficiency cores): taskpolicy -b command
    • Downgrade existing process to run in background: taskpolicy -b -p pid
    • Remove from background (efficiency or performance cores): taskpolicy -B -p pid
    • Start in suspended mode (useful for profiling/debuging): taskpolicy -s command
    • Get info on the scheduling policy: sudo taskinfo pid
  • Text to speech engine: say
  • Power management:
    • Put the display in sleep: pmset displaysleepnow
    • Put the Mac in sleep: pmset sleepnow
  • Manage macOS updates: softwareupdate
  • Network settings: networksetup
    • IPv6 activation: networksetup -setv6off Wi-Fi / networksetup -setv6automatic Wi-Fi
  • Wi-Fi: wdutil
    • Get information: sudo wdutil info
    • Diagnosis: sudo wdutil diagnose
  • System Profiling: system_profiler

System (mainly GNU/Linux)

  • Process’s system calls: strace -pPID -f
    • with -e open zsh listing opened files
    • with -e file zsh listing read/write
  • Global memory stats: vmstat or /proc/meminfo
  • process’s memory: pmap PIDor /proc/PID/smaps -status
  • Processor: trace, top, htop
  • Kernel’s messages: dmesg
  • Disk Usage: du -Psckxh * | sort -nr
  • See details of an app’s virtual size: vmmap -resident PID# | grep "Summary for" -A 999

Networking

  • Tunnel ssh, connecting through remote_server to reach remote_remote_server: ssh -L 666:remote_remote_server:9999 -l root remote_server -N
  • Testing a connection and prompt: telnet address port
  • Testing a connection: nc -v address port
  • Resolution name: nslookup domain_name
  • Domain owner information: whois domain_name
  • Trying getting a name from an IP: dig -x IP
  • Monitoring network: tcpdump ; or GUI: Wireshark
  • All connections: netstat --tcp --extend --progress
  • Activity: nettop

For Developers

  • Extracting links from HTML (file or online page: lynx -dump -listonly my.html
  • Difference between two GIT commits: git log --format=format:"%h, %ci %an, %s" 05c3bfc7..f35b951
  • Converting Markdown to Microsoft Word: 
    INPUT_ADOC=my_input_file.adoc
asciidoctor --backend docbook --out-file - $INPUT_ADOC|pandoc --from docbook --to docx --output $INPUT_ADOC.docx
  • Java - extracting Exceptions or ERROR from logs: grep -P " (ERROR|^\tat |Exception ^Caused by:.+ more)"
  • Using cookies for web authentification:
    • Getting cookies: wget --save-cookies cookies.txt --keep-session-cookies --post-data $AUTHENT --delete-after https://$URL
    • Using cookies: wget -v -e robots=off --load-cookies=cookies.txt $URL
  • Using JSON token for web authentification:
    • Getting token: TOKEN=$(curl -X POST "$URL/auth/login" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"username\":\"$USERNAME\",\"password\":\"$PASSWORD\"}" | cut -d':' -f8 | sed s/\"//g | sed s/\}//g)
    • Using token: curl -X GET "$URL" -H "accept: application/json" -H "Authorization: Bearer $TOKEN"
  • Splitting file in files: csplit FILE PATTERN
  • Rendering Markdown files:
    • Web: pandoc file.md | lynx -stdin
    • Plain text: pandoc -t plain file.md | less

Utils

  • Getting file’s information: file the_file
  • Backup a directory to another one: rsync -azv --delete --delete-excluded --exclude='no backup/' /path/to/source /path/to/backup
  • Transferring large files (handling partial download): rsync -P -e "ssh --i identity_key" user@remote_server:file_dest .
  • Ruuning a command every 2s: watch