running ssh workflow on group of servers

#generate sample servers

for i in {1..110}; do printf "myserver%05d\n" $i; done > myservers.txt

#group servers by 20
count=0
group=0
for line in $(cat  myservers.txt);
do
  printf "GROUP_%03d %s\n" $group $line
  ((count=$count + 1))
  if [ $count -eq 20 ]; then
    ((group=$group + 1))
    count=0
  fi
done >> grouped.txt

#print servers in given group
cat grouped.txt | grep GROUP_005 | awk -F' ' '{print $2}'


#put this in steps.sh

myserver=$1
step=$2


case $step in
  1)
    echo "hello, this is step 1 for server $myserver"
    ;;
  2)
    echo "ciao, this is step 2 for server $myserver"
    ;;
  3)
    echo "Gruezi, this is step 3 for server $myserver"
    ;;
  *)
    echo "ERROR invalid step $step"
esac

#execute given step for GROUP

THESTEP=3; cat grouped.txt | grep GROUP_005 | awk -vstep="$THESTEP" -F' ' '{print $2,step}' | xargs ./steps.sh

Audit the content of a series of folders against a file

the audit.txt contains the list of original files:

/media/sf_shared/bashtests/dirtoaudit/
/media/sf_shared/bashtests/dirtoaudit/dir01
/media/sf_shared/bashtests/dirtoaudit/dir01/file01_01.txt
/media/sf_shared/bashtests/dirtoaudit/dir01/file02_01.txt
/media/sf_shared/bashtests/dirtoaudit/dir02
/media/sf_shared/bashtests/dirtoaudit/dir02/file01_02.txt
/media/sf_shared/bashtests/dirtoaudit/dir02/file02_02.txt

this script checks that in the folders

/media/sf_shared/bashtests/dirtoaudit/
/media/sf_shared/bashtests/dirtoaudit/dir01
/media/sf_shared/bashtests/dirtoaudit/dir02

there are no extra files or folders:




Of course this scales very poorly... I would never dream of writing complex logic in bash, unless I was absolutely forced

bash comparison and validation of string

trying to understand Bash syntax is really wasted time.... just copy/paste working examples

array=("pippo pluto topolino")
value=pluto

[[ " ${array[@]} " =~ " ${value} " ]] && echo "YES" || echo "NO"

if [[ " ${array[@]} " =~ " ${value} " ]]; then echo trovato; fi

pippo="ciao"
[[ $pippo = "ciao" ]] && echo "1yes"
[[ "ciao" = "ciao" ]] && echo "2yes"

x="valid"
if [ "$x" = "valid" ]; then
  echo "x has the value 'valid'"
fi

[[ "$x" = "valid" ]] && echo "x is valid" 

[ "$x" == "valid" ] && echo "x has the value 'valid'"

[ "$x" == "valid" ] && echo "i am valid" || echo "i am invalid"

shell testing

I have never seen in my life a bash shell being covered by automated tests.

I have thought of using Java and Mockito and Junit5, but it's not very straightforward to run shells from Java (in 2019.... maybe in 2 years it will be normal).

But I think it would be an excellent idea.

This is an inspiring article https://www.leadingagile.com/2018/10/unit-testing-shell-scriptspart-one/

This is the shunit2 framework:

https://github.com/kward/shunit2/


Here the reference manual for shell scripting http://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html but it's a bit too academic.

https://www.tldp.org/LDP/abs/html/index.html this one is richer of examples

PS shell scripting sucks

bash script cheatsheet

debug hanging shell scripts

https://stackoverflow.com/questions/4640794/a-running-bash-script-is-hung-somewhere-can-i-find-out-what-line-it-is-on

set -x

lsof -p $pid

strace -p $pid -s 1024

/proc/$pid/environ

pstree -pl 21156 + cat /proc/15232/stack (15232 is the PID of the hanging process)

top

set -v or set -o verbose

http://bashdb.sourceforge.net/
https://www.shellcheck.net/
https://marketplace.visualstudio.com/items?itemName=rogalmic.bash-debug

Deleting a non empty directory with find -delete

mkdir  /opt/oracle/Pipposcriptsstaging/PippoScripts-1.23
mkdir  /opt/oracle/Pipposcriptsstaging/PippoScripts-1.22

(put some content in both folders)

this will fail:

find /opt/oracle/Pipposcriptsstaging/  -type d -name "PippoScripts*" ! -name "PippoScripts-1.22" -delete


because find -delete can only remove empty folders (don't try the -maxdepth, -depth, -mindepth, -prune options, they won't work)


For some REALLY weird reason, this command too will fail:

find /opt/oracle/Pipposcriptsstaging/  -type d -name "PippoScripts*" ! -name "PippoScripts-1.22" -exec rm -rf '{}' \;
find: `/opt/oracle/Pipposcriptsstaging/PippoScripts-1.23': No such file or directory

but it actually deletes the PippoScripts-1.23 folder (WEIRD!)


The only way to make it work:

find /opt/oracle/Pipposcriptsstaging/PippoScripts-1.23 -type f  -delete

find /opt/oracle/Pipposcriptsstaging/PippoScripts-1.23 -type d  -delete

bash: redirect stdout and stderr for a block of "code"

(I was hesitant to call "code" a bash script, as it's mostly molasses of un-refactorable hieroglyphs)

vi redirtest.sh

{
plutti
plitti
} > plutti.log 2>&1

{
pippi
} > pippi.log 2>&1


{
echo ciao
} > ciao.txt

{
echo miao
} > miao.txt

The command "plutti" and "plitti" don't exist, so I will have an error "./redirtest.sh: line 2: plutti: command not found ./redirtest.sh: line 2: plitti: command not found".
But only that block will be redirected to plutti.log.
Same story for pippi.log: you have 2 separate error logs for the 2 blocks of code.
So, not necessarily redirection has to be at the whole script level, or at the single statement level.... one can group several statements in a "try/catch" block, which is cool...IMHO at least, it gives more flexibility ...

bash scripts, success and failure

All this might be trivial, but it's good to have it written down somewhere.


false and true can be used to simulate a command which failed or succeeded(respectively)

false
echo $?
1
true
echo $?
0

You can use || to run a command only if the previous failed:

false || echo "it failed"

it failed 

true || echo "it didn't fail"

(nothing is printed here) 

You can use && to run a command only if the previous succeeded:

true && echo "It succeeded"

It succeeded

false && echo "It succeeded"

(nothing is printed here)  

You can chain as many command as you want with && and || (normally you do this only with &&).

set -e  will make sure your script stops execution as soon as a command returns something != 0

adding a timestamp to bash stdout

Assume a shell script writes without timestamp info:

myscript.sh
bla 
mumble

how long it took to complete bla? and mumble? no clue. You could hack the script to add a timestamp for each echo, but that's not practical. Workaround:

vi timestamp.sh

#! /bin/bash

while read line
do
echo `date +"%Y%m%d-%H%M%S"` $line
done

chmod 775 timestamp.sh

now you can use it this way:

myscript.sh | ./timestamp.sh

20140415-145529 bla
20140415-145529 mumble

In alternative, you can also compile this c program:

//gcc timestamplog.c -o timestamplog

#include 
#include 

time_t now;
struct tm  *ts;
char buf[80];

void printTime() {
    now = time(NULL);
    ts = localtime(&now);
    strftime(buf, sizeof(buf), "%Y%m%d-%H%M%S  ", ts);
    printf("%s", buf);
}


int main() {
 int startLine = 1;
 int ch;
    while ((ch = getchar()) != EOF) {
  if (startLine) {
   printTime();
   startLine = 0;
  }
  
  if (ch == 10 || ch == 13) {
      //putchar('\n');
      putchar(ch);
      startLine = 1;

  } else {
         putchar(ch);
  }
  fflush(stdout);
    }

    return 0;
}

Fail fast shell scripts with set -e

Shell script number one:
vi testsete.sh

ls /pippo
ls /pappo

chmod 775 testsete.sh
./testsete.sh

ls: /pippo: No such file or directory
ls: /puppo: No such file or directory

now let's insert a "set -e" at the top of the script, and run again:

ls: /pippo: No such file or directory

See? Listing an nonexistent directory returns a non-zero error code, and this makes the script fail immediately, no need for boring explicit testing of the exit code of each command.... I only wish there was an error handler to print a nice error message.... I guess 100 years from now bash will evolve to something decent, for the time being be happy with the horrible crap it is.

find files with windows style linefeed

find . -name "*.sh" -exec grep -l "\r\n" {} \;

to fix them:

find . -name "*.sh" -exec grep -l "\r\n" {} \; | xargs dos2unix

Bash: reading properties based on an ENV parameter

I hate bash, and I hate that there is not a standard way of reading property files based on 2 parameters: the property name, and the environment. And we are in 2014...flying people to Mars etc...and still hacking these very basic requirements.

Here is a possible solution:

create a config.sh executable file:

DEV_prop1=bla
PROD_prop1=blu

you have your environment name in a variable ENV

ENV="DEV"

you load all your properties:

. ./config.sh  (note the space after the first . !)

then you get the property name with an "indirect referencing":

propertyname=${ENV}_prop1
myvalue="${!propertyname}";

There are many other ways (I hate the cat/grep/awk solution) but they all stink just the same.
Let's face it, bash stinks. Like a skunk.

bash scripts organized with rerun

A small "rerun tutorial" or "getting started with rerun".
Rerun wiki: https://github.com/rerun/rerun/wiki

Download the rpm here

#remember to do export http_proxy=http://myuser:mypw@proxy.acme.com:8080

wget http://repository-stagrlee.forge.cloudbees.com/release/rpms/rerun/rerun-1.0.2-1.fc17.noarch.rpm
rpm -i rerun-1.0.2-1.fc17.noarch.rpm
rerun Available modules in "/usr/lib/rerun/modules":
stubbs: "Simple rerun module builder" - 1.0.2


then I download https://github.com/downloads/rerun-modules/rerun-modules/rerun-modules-repo-1.0-21.noarch.rpm, copy it locally (wget doesn't work with https on github, no clue...)
and I do:

rpm -i rerun-modules-repo-1.0-21.noarch.rpm

this one didn't work for me (connectivity problems) but it's not essential:
yum -y --disablerepo '*' --enablerepo 'rerun-modules' install '*'


I create my first module

rerun stubbs: add-module
Module name:
pippo
Module description:
my first rerun
Created module structure: /usr/lib/rerun/modules/pippo.


If you look in /usr/lib/rerun/modules/pippo, there is a lib folder - containing an empty functions.sh - and a command folder.

rerun stubbs:add-command

Module:
1) pippo
2) stubbs
#? 1
You picked module pippo (1)
Command:
hello
Description:
say hello
Wrote command script: /usr/lib/rerun/modules/pippo/commands/hello/script
Wrote test script: /usr/lib/rerun/modules/pippo/tests/hello-1-test.sh

then I add an option to the command:
rerun stubbs:add-option

Module:
1) pippo
2) stubbs
#? 1
You picked module pippo (1)
Command:
1) hello
#? 1
You picked command hello (1)
Option:
name
Description:
the person to greet
Required:
1) true
2) false
#? 1
Export:
1) true
2) false
#? 1
Default:
Pierre
Wrote option metadata: /usr/lib/rerun/modules/pippo/options/name/metadata
Updated command metadata:  /usr/lib/rerun/modules/pippo/commands/hello/metadata
Updated command script header: /usr/lib/rerun/modules/pippo/commands/hello/script

vi /usr/lib/rerun/modules/pippo/commands/hello/script
and after this text:

# - - -
# Put the command implementation here.
# - - -

I enter this: echo name is $NAME

I run it:

rerun pippo:hello --name Luigi
name is Luigi

and rerun it:

rerun pippo:hello
name is Pierre

Ok at least we have proven that it can can be made to work...

bash for loop

just a quick note, I always forget the syntax:

for i in {1..5}; do wget --no-check-certificate  https://mserver.com:8002/blatest; done

String manipulation in Bash shell

I am really fed up by people who use unreadable sed and awk commands to perform basic string manipulations, like a find and replace on a string.

There are very nice expression you can use:

http://tldp.org/LDP/abs/html/string-manipulation.html

string length: ${#string}

expression matching: expr match "$string" '$substring'

index: expr index $string $substring

substring: ${string:position:length}

substring delete: ${string#substring}

find and replace: ${string//substring/replacement}


Please use awk and sed only if REALLY needed.

The only good code is readable code.

Bash script to find the location of the currently running script

#!/bin/bash
#==========================
# bash - find path to script
#==========================
abspath="$(cd "${0%/*}" 2>/dev/null; echo "$PWD"/"${0##*/}")"

# to get the path only - not the script name - add
path_only=`dirname "$abspath"`

#display the paths to prove it works
echo $abspath
echo $path_only

(not my script, I have copied from here http://forums.macosxhints.com/archive/index.php/t-73839.html )


Otherwise, this is much simpler:

SCRIPT=`readlink -f $0`
SCRIPTPATH=`dirname $SCRIPT`
echo $SCRIPTPATH

Shell Script to build a classpath dynamically with all jars in a directory

copied from http://twit88.com/blog/2008/02/21/unix-shell-script-to-build-classpath-dynamically/ (thanks!)

#!/bin/sh

buildClassPath() {
jar_dir=$1
if [ $# -ne 1 ]; then
echo "Jar directory must be specified."
exit 1
fi
class_path=
c=1
for i in `ls $jar_dir/*.jar`
do
if [ "$c" -eq "1" ]; then
class_path=${i}
c=2
else
class_path=${class_path}:${i}
fi
done
echo $class_path
#return $class_path
}

CP=`buildClassPath /tmp/lib`
echo $CP

A simplified version:

COH_CLASSPATH=.:$SL_CACHE_CONF_DIR:$COHERENCE_HOME/lib/coherence.jar
for i in `ls $SL_CACHE_LIB_DIR/*.jar`
do
COH_CLASSPATH=${COH_CLASSPATH}:${i}
done