UNIX Commands

Homepage / Notes / Computer Science / UNIX Commands

UNIX Commands

rm

ReMoves a file

rm ~/tmp/unix_example

echo

Writes argument to new line of file

echo "yo" >> ~/tmp/unix_example

cat

Display the content of specified file

cat ~/tmp/unix_example

yo

sort

cat ~/tmp/sort_example

8
4
6
2
3
1
5
9
7

sort ~/tmp/sort_example

1
2
3
4
5
6
7
8
9

Works with pipe operator too:

cat ~/tmp/sort_example | sort

uniq

cat ~/tmp/uniq_example

2
1
3
3
2
2
2
1

cat ~/tmp/uniq_example | uniq

cat ~/tmp/uniq_example | sort | uniq

tail

Display the last part of a file

cat ~/tmp/sort_example | sort | tail -n 5

tail -f file: keep reading file until Ctrl-C

less

less file: interactively read file

less -F file equivalent to tail -f file

grep

Find patterns in files

cat ~/tmp/csv_example

product_id	product_name	url	price	currency
1	Nike shoes	example.com/nike	80	EUR
2	Adidas shoes	example.com/adidas	75	EUR
3	Puma shoes	example.com/puma	82	EUR
4	Burton bag	example.com/burton	55	EUR
5	Canada Goose parka	example.com/canada-goose	600	CAD

cat ~/tmp/csv_example | grep "Nike"

Nike shoes

example.com/nike

EUR

awk

https://www.gnu.org/software/gawk/manual/gawk.html https://earthly.dev/blog/awk-examples/ https://www.cs.princeton.edu/courses/archive/spring19/cos333/awk.help

cat ~/tmp/csv_example | awk -F ',' '{print $2}'

product_name
Nike shoes
Adidas shoes
Puma shoes
Burton bag
Canada Goose parka

cat ~/tmp/csv_example | awk -F ',' '{print $4}'

price

600

sed

sed 's/example/amazon/' ~/tmp/csv_example

product_id	product_name	url	price	currency
1	Nike shoes	amazon.com/nike	80	EUR
2	Adidas shoes	amazon.com/adidas	75	EUR
3	Puma shoes	amazon.com/puma	82	EUR
4	Burton bag	amazon.com/burton	55	EUR
5	Canada Goose parka	amazon.com/canada-goose	600	CAD

cut

cat ~/tmp/csv_example | cut -d',' -f3

url

example.com/nike

example.com/adidas

example.com/puma

example.com/burton

example.com/canada-goose

tr

cat ~/tmp/csv_example | cut -d',' -f3 | tr / ?

url

example.com?nike

example.com?adidas

example.com?puma

example.com?burton

example.com?canada-goose

ncdu

Faster implementation for SSDs, written in Go: gdu

reset

Reset comsol to its initial state

jq

https://stedolan.github.io/jq/manual/ jq file.json to parse JSON file

Pretty printing

echo '{"key1":{"key2":"value1"}}' | jq '.'

{
  "key1": {
    "key2": "value1"
  }
}

Selecting a specific key

curl https://api.github.com/repos/mewfree/mewfree | jq '.owner'

{
  "login": "mewfree",
  "id": 8095395,
  "node_id": "MDQ6VXNlcjgwOTUzOTU=",
  "avatar_url": "https://avatars.githubusercontent.com/u/8095395?v=4",
  "gravatar_id": "",
  "url": "https://api.github.com/users/mewfree",
  "html_url": "https://github.com/mewfree",
  "followers_url": "https://api.github.com/users/mewfree/followers",
  "following_url": "https://api.github.com/users/mewfree/following{/other_user}",
  "gists_url": "https://api.github.com/users/mewfree/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/mewfree/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/mewfree/subscriptions",
  "organizations_url": "https://api.github.com/users/mewfree/orgs",
  "repos_url": "https://api.github.com/users/mewfree/repos",
  "events_url": "https://api.github.com/users/mewfree/events{/privacy}",
  "received_events_url": "https://api.github.com/users/mewfree/received_events",
  "type": "User",
  "site_admin": false
}

curl https://api.github.com/repos/mewfree/mewfree | jq '.owner.html_url'

"https://github.com/mewfree"

Selecting elements from an array

echo "[1,2,3,4,5]" | jq '.[0]'

echo "[1,2,3,4,5]" | jq '.[0:2]'

[
  1,
  2
]

echo "[1,2,3,4,5]" | jq '.[2:]'

[
  3,
  4,
  5
]

echo "[1,2,3,4,5]" | jq '.[-2:]'

[
  4,
  5
]

Selecting keys from elements from an array

echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq '.[].title'

"a"
"b"
"c"

echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq '.[0].title'

"a"

Raw strings

echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq -r '.[0].title'

Join strings

echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq -j '.[].title'

abc

Array constructor

echo '[{"title": "a"}, {"title": "b"}, {"title": "c"}]' | jq '[ .[].title ]'

[
  "a",
  "b",
  "c"
]

Select multiple fields

By iterating:

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq '.[].type, .[].title'

"movie"
"book"
"tv show"
"The Social Network"
"Ready Player One"
"Halt And Catch Fire"

Using the pipe operator:

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq '.[] | .type, .title'

"movie"
"The Social Network"
"book"
"Ready Player One"
"tv show"
"Halt And Catch Fire"

Pipe operator wrapped in array constructor:

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq '[ .[] | .type, .title ]'

[
  "movie",
  "The Social Network",
  "book",
  "Ready Player One",
  "tv show",
  "Halt And Catch Fire"
]

Using the object constructor:

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq '[ .[] | {media_type: .type, title: .title } ]'

[
  {
    "media_type": "movie",
    "title": "The Social Network"
  },
  {
    "media_type": "book",
    "title": "Ready Player One"
  },
  {
    "media_type": "tv show",
    "title": "Halt And Catch Fire"
  }
]

Sort / Reverse / Length

echo '["3","2","1"]' | jq 'sort'

[
  "1",
  "2",
  "3"
]

echo '["1","2","3"]' | jq 'reverse'

[
  "3",
  "2",
  "1"
]

echo '["1","2","3"]' | jq 'length'

Map / Select

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq 'map({ title: .title })'

[
  {
    "title": "The Social Network"
  },
  {
    "title": "Ready Player One"
  },
  {
    "title": "Halt And Catch Fire"
  }
]

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq 'map(select(.type == "tv show")) | map({title: .title})'

[
  {
    "title": "Halt And Catch Fire"
  }
]

echo '[{"type": "movie", "title": "The Social Network", "release_date": "2010"}, {"type": "book", "title": "Ready Player One", "release_date": "2011"}, {"type": "tv show", "title": "Halt And Catch Fire", "release_date": "2014"}]' \
    | jq 'map(select(.type != "book")) | map({title: .title, date: .release_date})'

[
  {
    "title": "The Social Network",
    "date": "2010"
  },
  {
    "title": "Halt And Catch Fire",
    "date": "2014"
  }
]

fx

fx file.json to view and interact with JSON file

z

Alternative cd? https://github.com/rupa/z

tree

Display directory structure

tree -I .git ignore .git tree -a show "hidden" files

stow

https://alexpearce.me/2016/02/managing-dotfiles-with-stow/ stow vim creates a symlink between files in vim and upper directory

shuf

"shuffle"

seq 10 | shuf

9
6
7
1
10
5
8
3
2
4

seq 10 | shuf -n 1

dasel

https://github.com/TomWright/dasel jq for JSON, YAML, TOML, XML and CSV

Querying for a specific field

echo '{"id": 1, "name": "Damien"}' | dasel -r json '.name'

"Damien"

JSON => CSV

echo '{"id": 1, "name": "Damien"}' | dasel -r json -w csv

id,name
1,Damien

CSV => JSON

printf 'id,name\n1,Damien\n2,Emilie' | dasel -r csv -w json

{
  "id": "1",
  "name": "Damien"
}
{
  "id": "2",
  "name": "Emilie"
}

Miller

https://miller.readthedocs.io/en/latest/index.html Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON. You get to work with your data using named fields, without needing to count positional column indices.

"cat" (unmodified)

printf 'id,name\n1,Damien\n2,Emilie' | mlr --csv cat

id,name
1,Damien
2,Emilie

Pretty print

printf 'id,name\n1,Damien\n2,Emilie' | mlr --csv --opprint cat

id name
1  Damien
2  Emilie

Select column(s)

printf 'id,name\n1,Damien\n2,Emilie' | mlr --csv cut -f name

name
Damien
Emilie

htmlq

https://github.com/mgdm/htmlq

curl --silent https://www.damiengonot.com/blog | htmlq --text article h3 a

🧨 Supercharge Your Spreadsheets With Free Emailing 📮
How to Automatically Pull Facebook Ads Data in Google Spreadsheet?
Guide to Facebook Insights API (Part 2)
Guide to Facebook Insights API
Getting Started with Facebook Marketing API
Connecting Google Ads Scripts with Google Spreadsheets
Google Ads Scripts: Programmatically Create Ads
Google Ads Scripts: Modify Entities
Google Ads Scripts: Reading Data
A Brief Introduction to Google Ads Scripts

pup

https://github.com/ericchiang/pup

curl --silent https://www.damiengonot.com/blog | pup 'article h3 a text{}'

ðŸ§¨ Supercharge Your Spreadsheets With Free Emailing ðŸ“®
How to Automatically Pull Facebook Ads Data in Google Spreadsheet?
Guide to Facebook Insights API (Part 2)
Guide to Facebook Insights API
Getting Started with Facebook Marketing API
Connecting Google Ads Scripts with Google Spreadsheets
Google Ads Scripts: Programmatically Create Ads
Google Ads Scripts: Modify Entities
Google Ads Scripts: Reading Data
A Brief Introduction to Google Ads Scripts

No support for emojis?

ufw

"UncomplicatedFireWall"

/!\ allow SSH when setting up ufw on a remote server so you don't lose the connection sudo ufw allow ssh

Status: sudo ufw status Enable: sudo ufw enable

Deny incoming but allow outgoing: sudo ufw default deny incoming sudo ufw default allow outgoing

csvquote

https://github.com/dbro/csvquote Enables common unix utlities like cut, awk, wc, head to work correctly with csv data containing delimiters and newlines

rsync

rsync {origin} {destination}

Works over ssh too: rsync user@host:/home/user /home/user/backup

Common arguments:

-r for recursive
-v for verbose (display file names)
-P to keep files partly transferred
-a archive mode, bundles a bunch of common arguments together for archives/backups
-h more human readable numbers
--progress to show progress during transfer

ffmpeg

Crop video

Live Preview

Crop bottom 100px from source.mp4: ffplay -i source.mp4 -vf "crop=in_w:in_h-100:0:0"

Actual

ffmpeg -i source.mp4 -filter:v "crop=in_w:in_h-100:0:0" -c:a copy destination.mp4

try

https://github.com/binpash/try

try lets you run a command and inspect its effects before changing your live system.

Help

Traditionally, can use man {command} to get info about command but can be quite lengthy and lack examples. tldr {command} solves this problem by being a short cheatsheet with many examples.

Resources

6 Command Line Tools for Productive Programmers

https://earthly.dev/blog/command-line-tools/

https://news.ycombinator.com/item?id=27992073

Command Line Interface Guidelines

https://clig.dev/