As of 2016-02-26, there will be no more posts for this blog. s/blog/pba/
Showing posts with label scripting. Show all posts

Note

This post is about parsing technique in Bash. If you have no idea what Bash is, then this is not the post for you.

If the following code is all you have done with Bash read builtin command, then you have missed a lot of fun!

read -p "Please enter your username: " username

Ever tried with -a, -n, or -t? At least, you have used -s (silent mode) for password input, right? I only used read to get user inputs. But someday, I found out it could be even more. Anyway, this post is about parsing, so those options are not the superstars today.

1Example

Say you want to get the process ID, which uses the most CPU resource currently, as well as the percentages of CPU utilization and memory usage.

% ps -u $UID -o pid= -o %cpu=,%mem= --sort -%cpu | line
  509 10.4  1.2

I wanted to see how much memory Firefox 4 use, not for any particular reason. So, I quickly wrote one short script:

#ff_pid=1234
max_mem=2048
max_bar_len=100
while :; do
ff_pid=$(ps -e -o pid,cmd | grep firefox-bin$ | grep -o '^ *[0-9]*')
if [[ $ff_pid ]]; then
  mem_mb=$(($(ps -p $ff_pid -o rss=)/1024))
  bar_len=$((max_bar_len*mem_mb/max_mem))
  printf "$(date +%H:%M) %4dM [%3d%%] \033[41m%${bar_len}s\033[42m%$((max_bar_len-bar_len))s\033[0m\n" $mem_mb $((100*mem_mb/max_mem)) '' ''
else
  printf "%s ---\n" $(date +%H:%M)
fi
sleep 60
done | tee ff_mem.txt | awk '{if(m!=$2){print $0;m=$2}}'

max_mem specifies the system memory in MB. max_bar_len is the how long the bar can be in character if memory use is 100%. Because I always run additional Firefox instance (profile, using --no-remote), so I have to make sure I get the main Firefoxs PID, which is the one I want to monitor. The awk is used for removing consecutive lines of same amount memory use. You can take it out if you want to see minute by minute results.

Here is a result after I started up Firefox today:

http://farm6.static.flickr.com/5081/5353295127_55ef2c1e6f_b.jpg

I havent fully used Firefox to visit sites I often do. Normally, after it reaches about 20%, I restart Firefox. I want to see if it will go down after it reaches 50%.

I wanted to know how I could monitor file system changes, especially activities on my home directory, and this was about this thread. I was thinking the issue could be solved by generating a snapshot of files using find, then watching changes and making changes to the snapshot file. Using grep to do the search.

But, thats simply stupid. Just using find would do that OP wants since that program OP uses only search for file names and folder names. For the first run of find it might be slower, 31000 more files took about six seconds. Well its not really slow. The consecutive runs only take about 0.1 seconds even after you add or delete files.

Anyway, its still fun to know how to monitor files. I installed inotify-tools package for inotifywait program.

http://farm6.static.flickr.com/5087/5205625460_60c806f25a.jpg

Here is the script I use:

inotifywait -m -r --format $'%T %e %w%f' --timefmt '%H:%M:%S' --exclude ~/'(\.mozilla|Documents/KeepNote)' -e modify -e move -e create -e delete ~ 2>&1 | awk '/^[0-9]/ {
sub(/'"${HOME//\//\\/}"'/, "~", $0)
split($0, a, " ")
len=length(a[1])+length(a[2])+1
printf "%-20s %s\n", substr($0, 0, len), substr($0, len+2)
// flush stdout
system("")
next
}
{print ; system("")}
' | tee -a /tmp/home_monitor

-m and -r are for recursively monitoring on files. I set up the output format and timestamp format. I exclude two things from being listed, one is ~/.mozilla and another is KeepNotes files. You can only have one --exclude supplied, if you have more than one, then only the last one would be effective. You have to group them up. -e specify events you want to monitor.

I use awk to do format adjustment, column alignment. Also replacing the literal of your home directory, /home/username, with just ~. Then I pipe the output to tee, so I can see on screen and write output to file at the same time. This way, Conky can ${tail /tmp/home_monitor 10} the file.

I was planning to use named pipe:

% MPIPE=/tmp/home_monitor
$ mkfifo $MPIPE
$ inotifywait ... | awk ... >$MPIPE
$ tail -f $MPIPE

It will save some disk space, but it doesnt work with Conky.

If you want to clean up /tmp/home_monitor when it gets too big, just do echo '' > /tmp/home_monitor. An important note for you, when you tail a file in Conky, make sure it exists. If you remove the file, Conky exits. I was hoping there was a Conky variable would run program in a thread and do similar task as $tail does, only the input is the standard output of that thread not a file.

By the way, I had never thought even in my own home directory would have many activities.

pee is a C code from moreutils. You can do something with it like:

% pee bc bc
1+2
1+2
[Press Ctrl+D]
3
3

$ echo 1+2 | pee bc bc
3
3

pee accepts arguments as commands. It will feed those commands with the standard input it gets.

As usual for moreutils post, I wrote a Bash script, pees:

#!/bin/bash

if (($# == 0)); then
  echo "Please specify command(s) as argument(s) of $(basename "$0")."
  exit 1
fi

TMPFILE="$(mktemp)"

while read line; do echo "$line" >> "$TMPFILE"; done

for cmdargs in "$@"; do bash -c "$cmdargs" < "$TMPFILE"; done

rm "$TMPFILE"

Needless to say, pees can do the examples above. But it can also do these as pee can:

ts is a Perl script from moreutils. You use it with pipe, it receives stdout of another program. The basic idea is doing:

command | ts

Its useful when you need to know when a program processes something. But its not enough for me, I wrote a Bash script ts.sh to do more.

#!/bin/bash

DATECMD='date +%H:%M:%S'

process_stdout() {
  while read line; do
    echo -e "\e[44m$($DATECMD)\e[0m $line"
  done
}

process_stderr() {
  while read line; do
    echo -e "\e[41m$($DATECMD)\e[0m $line" 1>&2
  done
}

if (( $# == 0 )); then
  process_stdout
else
  exec 3>&1
  ( bash -c "$*" | process_stdout ) 2>&1 1>&3 | process_stderr
  exec 3>&-
fi

It can timestamp stdout and stderr of a command with different color. See the following example:

http://farm2.static.flickr.com/1380/5160212248_15b5bf58b7.jpg

The timestamps on red is the stderr of emerge, the rest on blue is the stdout. This script could also be used with pipe, but it would only timestamp on stdin, which is a commands stdout via pipe. Its best to use it like:

ts.sh command arg1 arg2 ...

An example with shell script as command:

ts.sh while true \; do echo tik tok \; sleep 1 \; done

chronic is a Perl script from moreutils collection, runs a command quietly. It eats up the output of the command, but if the command exits with error, which is exit status does not equal to zero, then chronic would show you the output.

From its manpage, it provides a clear example how it could be useful with cron:

0 1 * * * chronic backup # instead of backup >/dev/null 2>&1

Note

I use vixin cron, I dont need to redirect stderr to stdout, just chronic backup >/dev/null would do in the old way.

The old way, using redirection is fine, would not give you message sent to stdout since they are redirected to /dev/null, thought you would see get stderr message from log or email if you have sent up.

chronic would get you both.

I made a simple Bash script, chronic.sh:

Sometimes, your script may need to accept a command and run that command in your script. I decided to take a look at $* and $. From manpage:

Special Parameters

* Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, it expands to a single word with the value of each parameter separated by the first character of the IFS special variable. That is, "$*" is equivalent to "$1c$2c...", where c is the first character of the value of the IFS variable. If IFS is unset, the parameters are separated by spaces. If IFS is null, the parameters are joined without intervening separators.

Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word. That is, "$" is equivalent to "$1" "$2" ... If the double-quoted expansion occurs within a word, the expansion of the first parameter is joined with the beginning part of the original word, and the expansion of the last parameter is joined with the last part of the original word. When there are no positional parameters, "$" and $ expand to nothing (i.e., they are removed).

I wrote a simple script to understand better:

#!/bin/bash

echo "\$# = $# arguments"
echo

TMP=($*) ; echo -n "(\$*)   => ${#TMP[]} elements => "
for arg in ${TMP[]}; do echo -n "$arg, " ; done ; echo # or in $*

TMP=("$*") ; echo -n "(\"\$*\") => ${#TMP[]} elements => "
for arg in "${TMP[]}"; do echo -n "$arg, " ; done ; echo # or in "$*"

TMP=($) ; echo -n "(\$)   => ${#TMP[]} elements => "
for arg in ${TMP[]} ; do echo -n "$arg, " ; done ; echo # or in $

TMP=("$") ; echo -n "(\"\$\") => ${#TMP[]} elements => "
for arg in "${TMP[]}" ; do echo -n "$arg, " ; done ; echo # or in "$"

echo
echo 'Running:'
"$"
% ./test.sh echo abc '123 456' def
$# = 4 arguments

($*)   => 5 elements => echo, abc, 123, 456, def,
("$*") => 1 elements => echo abc 123 456 def,
($)   => 5 elements => echo, abc, 123, 456, def,
("$") => 4 elements => echo, abc, 123 456, def,

Running:
abc 123 456 def

"$" is the one to use.

If I need to run a short shell script:

% ./test.sh bash -c 'for((i=0;i<3;i++)); do echo -n "$i," ; done ; echo'
$# = 3 arguments

($*)   => 11 elements => bash, -c, for((i=0;i<3;i++));, do, echo, -n, "$i,", ;, done, ;, echo,
("$*") => 1 elements => bash -c for((i=0;i<3;i++)); do echo -n "$i," ; done ; echo,
($)   => 11 elements => bash, -c, for((i=0;i<3;i++));, do, echo, -n, "$i,", ;, done, ;, echo,
("$") => 3 elements => bash, -c, for((i=0;i<3;i++)); do echo -n "$i," ; done ; echo,

Running:
0,1,2,

Or using bash -c "$*":

% ./test.sh echo abc '123 456' def
% ./test.sh 'for((i=0;i<3;i++)); do echo -n "$i," ; done ; echo'

https://i.ytimg.com/vi/pLb8AfLbkPo/mqdefault.jpg

Cat loves sponge as you can see in this video, but if its soaked with water, a sponge bathing? They would run away next time they see you hiding a sponge behind you. This video provides a nice instructions, if you want to try this one-time only bathing technique:

  1. Acquire cat and remove collar.

    First step is always the hardest, they have sixth sense, they can smell your intentions!

  2. Lightly Soap with dampened cloth (warm diluted solution of cat shampoo is best), ~ praise cat throughout ~

    I doubt they really buy it.

  3. Thoroughly rinse with warm damp cloth, then

  4. Gently towel dry. (do not iron)

    I bet some owners would want to try if ironing is more efficient.

I found its always fun to watch cat getting bathed, though there are some cat breeds do like water. And

Wait a minute!