Notas

Notas, scripts, algoritmos de uso diario y documentación de algunos proyectos.

View on GitHub
7 March 2021

Bash

by Juan Manuel González Garzón

Bash

I. Basic Linux Commands

A. Managing files and directories

# ls: lists the contents of the current directory, lists the contents of the received directory
ls [-la] $dirname
# chmod modifiers files: changes the permissions for the files according to the provided modifiers; we've seen +x to make the file executable
chmod
# chown user files: changes the owner of the files to the given user
chown
# chgrp group files: changes the group of the files to the given group
chgrp
# mkdir directory: creates the directory with the received name
mkdir $dirname
# cd directory: changes the current working directory to the specified one
cd $dirname
# pwd: prints the current working directory
pwd
# cp old_name new_name: copies old_name into new_name
cp
# touch file_name: creates an empty file or updates the modified time if it exists
touch
# mv old_name new_name: moves old_name into new_name
mv
# rmdir directory: deletes the directory with the received name (if empty)
rmdir $emptydir

B. Operating with the content of files

# cat file: shows the content of the file through standard output
cat $file
# wc file: counts the amount of characters, words, and lines in the given file; can also count the same values of whatever it receives via stdin
wc $file
# file file: prints the type of the given file, as recognized by the operating system
file $file
# head file: shows the first 10 lines of the given file
head $file
# tail file: shows the last 10 lines of the given file
tail $file
# less file: scrolls through the contents of the given file (press "q" to quit)
less $file
# sort file: sorts the lines of the file alphabetically
sort $file
# cut -dseparator -ffields file: for each line in the given file, splits the line according to the given separator and prints the given fields (starting from 1)
cut [options] [file]
cut -d [delimiter] -f [field number]

grep [pattern] [file-location]

C. Additional Commands

# echo "message": prints the message to standard output
echo "message"
# date: prints the current date
date +'%F'
# who: prints the list of users currently logged into the computer
who
# man command: shows the manual page of the given command; manual pages contain a lot of information explaining how to use each command (press "q" to quit)
man
# uptime: shows how long the computer has been running
uptime
# free: shows the amount of unused memory on the current system  
free
# ps: lists the processes executing in the current terminal for the current user
# ps ax: lists all processes currently executing for all users
# ps e: shows the environment for the processes listed  
ps [-ax] [e]
# kill PID: sends the SIGTERM signal to the process identified by PID
kill
# fg: causes a job that was stopped or in the background to return to the foreground
fg
# bg: causes a job that was stopped to go to the background
bg
# jobs: lists the jobs currently running or stopped
jobs
# top: shows the processes currently using the most CPU time (press "q" to quit)
top

grep
ping

II. Redirect IOStreams

# command > file: redirects standard output, overwrites file
# command >> file: redirects standard output, appends to file
# command < file: redirects standard input from file
# command 2> file: redirects standard error to file

# Redirection of STDOUT
echo 'lol1' > new_file.txt
cat > [file]
# Append of STDOUT
echo 'lol2' >> new_file.txt
cat >> [file]

# Redirection of STDIN
filter.py < new_file.txt

# Redirect STDERR to a separate files
filter.py < new_file.txt 2> error_file.txt

# Redirect STDERR to STDOUT and redirection of STDOUT
filter.py > /dev/null 2>&1

III. Pipes and pipelines

# command1 | command2: connects the output of command1 to the input of command2

# IO stream redirection using pipes
ls | less
cat sprider.txt | tr ' ' '\n' | sort | unique -c | sort -nr | head
cat kaiku.txt | ./capitalize.py
./capitalize.py < haiku.txt
grep

capitalize.py

#!/usr/bin/env python3

import sys

for line in sys.stdin:
	print(line.strip().capitalize())

IV. Signalling Processes

ping
# Ctrl-c SIGINT
# Ctrl-z SIGSTOP
fg # Restart stoped process

kill # SIGTERM
ps ax | grep 'ping'
kill #PID

V. Creating Bash Scripts

gather-information.sh

#!/bin/bash

line="---------------------------------"
echo "Starting at: $(date)"; echo $line

echo "UPTIME"; uptime; echo $line

echo "FREE"; free; echo $line

echo "WHO"; who; echo $line

echo "Finishing at: $(date)"

VI. Using Variables, user imput and Globs

#!/bin/bash

HERE=$PWD

function confirm() {
    read -p "Continue? (Y/N): " confirm && [[ $confirm == [yY] || $confirm == [yY][eE][sS] ]] || exit 0
}

if [ echo '=== 1. EMPAQUETANDO FRONT-END' && confirm ]; then
	cd Angular-Frontend
	ng build
	cd $HERE
fi

if [ echo '=== 2. VACIANDO LA CARPETA ESTATICA DE SPRING-BOOT' && confirm ]; then
	rm -rf "$HERE/Spring-Backend/src/main/resources/static/*"
	cd $HERE
fi

if [ echo '=== 3. MOVIENDO LA APLICACION DE ANGULAR DENTRO DE LA CARPETA ESTATICA DE SPRING-BOOT' && confirm ]; then
	mv $HERE/Angular-Frontend/dist/Angular-Frontend/* "$HERE/Spring-Backend/src/main/resources/static/"
	cd $HERE
fi

# Globs
echo *.py
echo ?????.py

VII. expr & test

Los comandos expr y test se utilizan para realizar operaciones y comparaciones respectivamente

expr

Operador Descripción
+ Suma
- Resta
\* Multiplicación
/ Division
% Modulo
Operador Descripción
= Igualdad
!= Diferente
> Mayor
>= Mayor o igual
< Menor
<= Menor o igual
Operador Descripción
| Or
& And 
#!/bin/bash
read -p 'Introduce un número ' value1
read -p 'Introduce otro número ' value2
answer1=$(expr $value1 \* $value2)
echo "Multiplican: $answer1"

read -p 'Introduce un número ' value1
read -p 'Introduce otro número ' value2
answer2=$(expr $value1 = $value2)
echo "Resultado: $answer2"

rdm=$(expr $RANDOM % 100)
echo $rdm"

test

Opción Descripción
-f 0 si el archivo existe y es regular
-s 0 si el archivo existe y su tamaño es mayor a 0
-r 0 si el archivo existe y tiene permisos de lectura
-w 0 si el archivo existe y tiene permisos de escritura
-x 0 si el archivo existe y tiene permisos de ejecución
-d 0 si el directorio existe
Opción Descripción
-lt Menor que
-le Menor o igual que
-qt Mayor que
-qe Mayor o igual que
-qa Igual a
-ne No igual a
-n True si el tamaño del string es non-zero.
Opción Descripción
-o Or
-a And
! Not 
#!/bin/bash
test -f /etc/passwd # 0
test [string1 = string2] # 1
test [string1 != string2] # 0
test 20 -lt 40 # 0

VIII. Conditional Execution

#!/bin/bash

# $? # exit status of command

read -p 'Introduce un número ' value
if (( value < 10 )); then
	echo 'número menor a 10'
else
	echo 'número mayor o igual a 10'
fi

if [[ grep "127.0.0.1" /etc/hosts ]]; then
	echo "OK"
else
	echo "Error! 127.0.0.1 is not in /etc/hosts"
fi

if test -n "$PATH"; then echo "Your path is not empty"; fi

# [] = alias of test command
if [ -n "$PATH" ]; then echo "Your path is not empty"; fi

IX. While Loops in Bash Scripts

#!/bin/bash

n=1
while [ $n -le 5 ]; do
	echo "Iteration number $n"
	((n+=1))
done

retry.sh

#!/bin/bash
n=0
commands=$1
while ! command && [ $n -le 5 ]; do
	sleep $n
	((n+=1))
	echo "Retry #$n"
done

# ./retry.sh ./random-exit.py

random_exit.py

#!/usr/bin/env python3

import system
import random

value=randomint(0, 3)
print("Returning: "+ str(value))
sys.exit(value)

X. For Loops in Bash Scripts

#!/bin/bash

PROCESS_DATES=(
'2016-04-11'
'2016-04-12'
'2016-04-13'
'2016-04-14'
'2016-04-15'
)

for PROCESS_DATE in "${PROCESS_DATES[@]}"; do
    echo "$PROCESS_DATE"
done

# Rename al the .HTM files into .html files.
for file in *.HTM; do
	name=$(basename "$file" .HTM)
	# echo is for precautions
	echo mv "$file" "$name.html"
done

for (( n=0 ; n < 5 ; n++ )) ; do echo "n: $n" ; done

XI. Advanced Command Interaction

tail /var/log/syslog | cut -d ' ' -f5-
cut -d ' ' -f5- /var/log/syslog | tr ' ' '\n' | sort | unique -c | sort -nr | head

tail /var/log/syslog | cut -d' ' -f3-

changeJane.py

#!/usr/bin/env python3

import sys

for line in sys.stdin:
	print(line.strip().replace('jane', 'jdoe'))

findJane.sh

#!/bin/bash
grep ' jane ' ../data/list.txt | cut -d ' ' -f3 > oldFiles.txt

changeFiles.sh

#!/bin/bash
# No hay gawk en esa maquina
cat oldFiles.txt | gawk  -v dest=$(./changeJane.py < $0)'{print "mv "$0" "dest}'

XII. ~/.bashrc

# User specific aliases and functions
# ====================================================================================
# alias unity3d='~/Downloads/Unity3D/UnityHub.AppImage & disown'

# Enable npm -g
export PATH=~/.npm-global/bin:$PATH

# DOTNET_ROOT
export DOTNET_ROOT=/snap/dotnet-sdk/current
export DOTNET_CLI_TELEMETRY_OPTOUT=1

# Install Ruby Gems to ~/gems
export GEM_HOME="$HOME/gems"
export PATH="$HOME/gems/bin:$PATH"

alias postman='/home/usuario/Postman-linux-x64-8.8.0/Postman/Postman & disown'

#COLORS
CDEF=" \033[0m"                                     # default color
CCIN=" \033[0;36m"                                  # info color
CGSC=" \033[0;32m"                                  # success color
CRER=" \033[0;31m"                                  # error color
CWAR=" \033[0;33m"                                  # waring color
b_CDEF=" \033[1;37m"                                # bold default color
b_CCIN=" \033[1;36m"                                # bold info color
b_CGSC=" \033[1;32m"                                # bold success color
b_CRER=" \033[1;31m"                                # bold error color
b_CWAR=" \033[1;33m"                                # bold warning color

BINARY_FILE_MATCHES='Coincidencia en el fichero binario'
function lookf() {
	grep -rni --exclude-dir={node_modules,.git,dist,testreports,build} "$@" ./ \
		grep -v "${BINARY_FILE_MATCHES}" \
		sort -u \
		grep "$@"
}

function lookfn() {
	grep -rli --exclude-dir={node_modules,.git,dist,testreports,build} "$@" ./ \
		grep -v "${BINARY_FILE_MATCHES}" \
		sort -u
}

function lookn() {
	find . \
		-not -path "*/node_modules/*" \
		-not -path "*/.git/*" \
		-not -path "*/dist/*" \
		-not -path "*/testreports/*" \
		-not -path "*/build/*" \
		-type f | grep -i "$@"
}

function lookna() {
	find "$(pwd -P)" \
		-not -path "*/node_modules/*" \
		-not -path "*/.git/*" \
		-not -path "*/dist/*" \
		-not -path "*/testreports/*" \
		-not -path "*/build/*" \
		-type f | grep -i "$@"
}

# echo like ...  with  flag type  and display message  colors
prompt () {
  case ${1} in
    "-s"|"--success")
      echo -e "${b_CGSC}${@/-s/}${CDEF}";;          # print success message
    "-e"|"--error")
      echo -e "${b_CRER}${@/-e/}${CDEF}";;          # print error message
    "-w"|"--warning")
      echo -e "${b_CWAR}${@/-w/}${CDEF}";;          # print warning message
    "-i"|"--info")
      echo -e "${b_CCIN}${@/-i/}${CDEF}";;          # print info message
    *)
    echo -e "$@"
    ;;
  esac
}

# Check command avalibility
function has_command() {
  command -v $1 > /dev/null
}

ACCIO_FOLDER='/home/usuario/accio-folder/'
ACCIO_SCRIPT='/home/usuario/accio-script.sh'
ACCIO_FILE='/home/usuario/accio-file.txt'
function accio() {
	if [ -d ${ACCIO_FOLDER} ]; then
		rm -rf ${ACCIO_FOLDER}
		mkdir ${ACCIO_FOLDER}
	else
		mkdir ${ACCIO_FOLDER}
	fi
	> "${ACCIO_FILE}"

	lookna $@ | \
		tee "${ACCIO_FILE}" | \
		gawk -v folder=${ACCIO_FOLDER} '{print "cp --backup=t \x27"$0"\x27 "folder}' | \
		tee "${ACCIO_SCRIPT}"

	sh "${ACCIO_SCRIPT}"
}

# latest: Lista los archivos en orden de creación o modificación mas reciente de manera global.
function latest() {
	find . \
		-not -path "*/node_modules/*" \
		-not -path "*/.git/*" \
		-not -path "*/dist/*" \
		-not -path "*/testreports/*" \
		-not -path "*/build/*" \
		-mtime 0
}


function prop {
  grep "^${1}=" ~/.workdir.jmgg | cut -d'=' -f2
}

function workdir() {
	if [ ! -z "$1" ]; then
		local work_dir=$(prop "${1}")
		cd ${work_dir}
	else
		local work_dir=$(prop 'work_dir')
		cd ${work_dir}
	fi
}

function workdirs() {
	cat ~/.workdir.jmgg
}

function setworkdir() {
	if [ ! -z "$1" ]; then
		# Erase last entry of the parameter on the file
		grep "${1}=" ~/.workdir.jmgg 2>&1 >/dev/null && sed -i "/${1}=/d" ~/.workdir.jmgg
		echo "${1}=${PWD}" >> ~/.workdir.jmgg
	fi
	# Erase again
	grep "work_dir=" ~/.workdir.jmgg 2>&1 >/dev/null && sed -i "/work_dir=/d" ~/.workdir.jmgg
	echo "work_dir=${PWD}" >> ~/.workdir.jmgg
}

function gadog() {
	git log --all --decorate --oneline --graph
}

function gdog() {
	git log --graph --oneline --decorate
}

function tree() {
	find . \
		-not -path "*/node_modules/*" \
		-not -path "*/.git/*" \
		-not -path "*/dist/*" \
		-not -path "*/testreports/*" \
		-not -path "*/build/*" \
		| sed -e "s/[^-][^\/]*\//  |/g" -e "s/|\([^ ]\)/|-\1/"
}

# ========== PS1 ==========
RED="$(tput setaf 1)"
GREEN="$(tput setaf 2)"
YELLOW="$(tput setaf 3)"
BLUE="$(tput setaf 4)"
PURPLE="$(tput setaf 5)"
CYAN="$(tput setaf 14)"
RESET="$(tput sgr0)"

exitstatus()
{
        local exitcode=$?
        if [[ $exitcode == 0 ]]; then
                echo ${GREEN}
        else
                printf '\a'
                echo ${RED}
        fi
        echo '('${exitcode}')'
}

thebranch()
{
        local branchname="$(__git_ps1)"
        if [[ ${branchname} != '' ]];  then printf "\n${CYAN}Branch:${branchname}${RESET}"; fi
}

PS1='$(exitstatus)'" [\A] \u${RESET}:${BLUE}\w${RESET} ["'$(ls -1 | wc -l | sed '"'s: ::g'"') elements, $(ls -lah | grep -m 1 total | sed '"'s/total //'"'), Jobs:\j, Users:$(who | wc -l)]$(thebranch)\n$'


# Busca ejecutables sospechosos:
# find /sbin /usr/sbin -executable \! -readable -print
# Search for files which are executable but not readable.

# zip old log files
# find $LOG_FOLDER -not -name "*.gz" -and -name "*.log" -mtime +90 -exec gzip {}

XIII. Bash Template

#!/bin/bash

USAGE="Usage: $0 [-a -u USER -H HOST ] [-b <parameter>] [-h]                    \n
	-a Execute action a                                                     \n
	-u user                                                                 \n
	-H host ip                                                              \n
	-b Excecute action b, expect <parameter>                                \n
	-h Show this help                                                       \n
"
# ============================== CONFIGURABLE ===========================
MAIN_ENV=true

# ============================== ATTRIBUTES =============================
EXECUTE_ACTION_A=false
VALUE_PARAMETER_B=
USER=
HOST=

# ============================== FUNCTIONS ==============================
function processInvocation () {
	# Check if there are options present
	while getopts "ab:u:H:h" opt; do
		case $opt in
			a) EXECUTE_ACTION_A=true;;
			b) VALUE_PARAMETER_B=${OPTARG};;
			u) USER=${OPTARG};;
			H) HOST=${OPTARG};;
			h) becho -e $USAGE
				exit 0
				;;
			*) recho -e $USAGE 1>&2
				exit 1
				;;
		esac
	done

	if [ $EXECUTE_ACTION_A = false ] || [ -z $USER ] || [ -z $HOST ] && [ -z $VALUE_PARAMETER_B ]; then
		recho -e $USAGE 1>&2
		exit 1
	fi

	# Example of parameters manipulation in a bash script, in function or global
	# becho '$@ Print all the parameters: '"$@"
	# becho '$0 Print the script name: :'"$0"
	# becho '$1 Print the first parameter: '"$1"
	# becho '$2 Print the second parameter: '"$2"
	# becho '$# Print the number of parameterS: '"$#"
}

function recho () {
	tput setaf 1;echo $@;tput sgr0
}

function gecho () {
	tput setaf 2;echo $@;tput sgr0
}

function becho () {
	tput setaf 6;echo $@;tput sgr0
}

function yecho () {
	tput setaf 3;echo $@;tput sgr0
}

function actionA () {
	yecho "Action A | Parameters: $@"
}

function debugInstruction () {
	set -x
	eval "$@"
	set +x
}

function main () {
	[ "$MAIN_ENV" = true ] && becho '========================================'

	if [ $EXECUTE_ACTION_A = true ]; then
		gecho 'Executing action A'
		gecho "USER=$USER"
		gecho "HOST=$HOST"
		debugInstruction actionA $USER $HOST
	fi

	if [ ! -z $VALUE_PARAMETER_B ]; then
		gecho "Executing action B with parameter: $VALUE_PARAMETER_B"
	fi

	[ "$MAIN_ENV" = true ] && becho '========================================'
}

# ============================== SCRIPT ==============================
processInvocation $@
main

# ./bash-template.sh -a -u Juan -H localhost

#!/bin/bash
# cProfile count how many times functions are called, and how long they run.
cProfile

# Copy files and directories recursively locally:
rsync -zrvh [Source-Files-Dir] [Destination]

List contents of directories in a tree-like format.

List with max display depth of the directory tree.

tree -L 2

List ignoring some folders

tree -I 'folder1|folder2'

List showing long name

tree -f

Search one word in a folder of documents

#!/bin/bash

TEXT_TO_SEARCH='TODO'

FOLDERS=(
	/PROJECTS/JAVA/LIB/ProyectName/
	/PROJECTS/JAVA/CLI/ProyectName/
	/PROJECTS/JAVA/WEB/ProyectName/
)

grep --include=\*.{java,sql,xml} -rnw -e "${TEXT_TO_SEARCH}" --exclude-dir={build,dist,.svn,node_modules,.git} "${FOLDERS[@]}"

List the heaviest 30 files in folder:

find . -type f -print0 | xargs -0 du | sort -n | tail -30 | cut -f2 | xargs -I{} du -sh {}

Creditos y referencias

Gnu bash iconos creados por Freepik - Flaticon

tags: