As of 2016-02-26, there will be no more posts for this blog. s/blog/pba/
Showing posts with label shell. Show all posts

Note

This post is about parsing technique in Bash. If you have no idea what Bash is, then this is not the post for you.

If the following code is all you have done with Bash read builtin command, then you have missed a lot of fun!

read -p "Please enter your username: " username

Ever tried with -a, -n, or -t? At least, you have used -s (silent mode) for password input, right? I only used read to get user inputs. But someday, I found out it could be even more. Anyway, this post is about parsing, so those options are not the superstars today.

1Example

Say you want to get the process ID, which uses the most CPU resource currently, as well as the percentages of CPU utilization and memory usage.

% ps -u $UID -o pid= -o %cpu=,%mem= --sort -%cpu | line
  509 10.4  1.2

I wanted to see how much memory Firefox 4 use, not for any particular reason. So, I quickly wrote one short script:

#ff_pid=1234
max_mem=2048
max_bar_len=100
while :; do
ff_pid=$(ps -e -o pid,cmd | grep firefox-bin$ | grep -o '^ *[0-9]*')
if [[ $ff_pid ]]; then
  mem_mb=$(($(ps -p $ff_pid -o rss=)/1024))
  bar_len=$((max_bar_len*mem_mb/max_mem))
  printf "$(date +%H:%M) %4dM [%3d%%] \033[41m%${bar_len}s\033[42m%$((max_bar_len-bar_len))s\033[0m\n" $mem_mb $((100*mem_mb/max_mem)) '' ''
else
  printf "%s ---\n" $(date +%H:%M)
fi
sleep 60
done | tee ff_mem.txt | awk '{if(m!=$2){print $0;m=$2}}'

max_mem specifies the system memory in MB. max_bar_len is the how long the bar can be in character if memory use is 100%. Because I always run additional Firefox instance (profile, using --no-remote), so I have to make sure I get the main Firefoxs PID, which is the one I want to monitor. The awk is used for removing consecutive lines of same amount memory use. You can take it out if you want to see minute by minute results.

Here is a result after I started up Firefox today:

http://farm6.static.flickr.com/5081/5353295127_55ef2c1e6f_b.jpg

I havent fully used Firefox to visit sites I often do. Normally, after it reaches about 20%, I restart Firefox. I want to see if it will go down after it reaches 50%.

I wanted to know how I could monitor file system changes, especially activities on my home directory, and this was about this thread. I was thinking the issue could be solved by generating a snapshot of files using find, then watching changes and making changes to the snapshot file. Using grep to do the search.

But, thats simply stupid. Just using find would do that OP wants since that program OP uses only search for file names and folder names. For the first run of find it might be slower, 31000 more files took about six seconds. Well its not really slow. The consecutive runs only take about 0.1 seconds even after you add or delete files.

Anyway, its still fun to know how to monitor files. I installed inotify-tools package for inotifywait program.

http://farm6.static.flickr.com/5087/5205625460_60c806f25a.jpg

Here is the script I use:

inotifywait -m -r --format $'%T %e %w%f' --timefmt '%H:%M:%S' --exclude ~/'(\.mozilla|Documents/KeepNote)' -e modify -e move -e create -e delete ~ 2>&1 | awk '/^[0-9]/ {
sub(/'"${HOME//\//\\/}"'/, "~", $0)
split($0, a, " ")
len=length(a[1])+length(a[2])+1
printf "%-20s %s\n", substr($0, 0, len), substr($0, len+2)
// flush stdout
system("")
next
}
{print ; system("")}
' | tee -a /tmp/home_monitor

-m and -r are for recursively monitoring on files. I set up the output format and timestamp format. I exclude two things from being listed, one is ~/.mozilla and another is KeepNotes files. You can only have one --exclude supplied, if you have more than one, then only the last one would be effective. You have to group them up. -e specify events you want to monitor.

I use awk to do format adjustment, column alignment. Also replacing the literal of your home directory, /home/username, with just ~. Then I pipe the output to tee, so I can see on screen and write output to file at the same time. This way, Conky can ${tail /tmp/home_monitor 10} the file.

I was planning to use named pipe:

% MPIPE=/tmp/home_monitor
$ mkfifo $MPIPE
$ inotifywait ... | awk ... >$MPIPE
$ tail -f $MPIPE

It will save some disk space, but it doesnt work with Conky.

If you want to clean up /tmp/home_monitor when it gets too big, just do echo '' > /tmp/home_monitor. An important note for you, when you tail a file in Conky, make sure it exists. If you remove the file, Conky exits. I was hoping there was a Conky variable would run program in a thread and do similar task as $tail does, only the input is the standard output of that thread not a file.

By the way, I had never thought even in my own home directory would have many activities.

For a long time, since I started to use Linux to be precise, I always thought I had a true Bourne shell installed until I did:

$ type sh
sh is /bin/sh
$ file /bin/sh
/bin/sh: symbolic link to `bash'

(actually, I did ls -l /bin/sh in second command.)

The truth is almost all of us do not have a Bourne shell. We are using a mimic by Bash (or Dash, if you use Ubuntu or Debian):

If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well. man bash

So, I asked Where the hell Bourne shell is? to myself. After some searches, I found The Traditional Bourne Shell Family. I downloaded two sources:

I tried some scripts on my system but I really couldnt find one would fail on two shells above. But, I wasnt really run those script since I didnt want to have unexpected files deletion or something like that.

So, I tried to focus finding side-effect when running Bash as sh. I read one clearly written in bash(1):

Brace expansion introduces a slight incompatibility with historical versions of sh. sh does not treat opening or closing braces specially when they appear as part of a word, and preserves them in the output. Bash removes braces from words as a consequence of brace expansion. For example, a word entered to sh as file{1,2} appears identically in the output. The same word is output as file1 file2 after expansion by bash. If strict compatibility with sh is desired, start bash with the +B option or disable brace expansion with the +B option to the set command (see SHELL BUILTIN COMMANDS below).

I tested:

$ sh -c 'echo file{1,2}' # Running bash as sh
file1 file2
$ ./sh -c 'echo file{1,2}' # Heirloom
file{1,2}
$ sh +B -c 'echo file{1,2}' # Turning off brace expansion
file{1,2}
$ dash  -c 'echo file{1,2}' # Dash
file{1,2}
$ ./v7sh  -c 'echo file{1,2}' # V7 shell
file{1,2}

Lets see what would happen if we intend to feed Bash sh with Bash syntax code:

$ bash -c 'for((i=0;i<3;i++)); do echo -n $i; done; echo'
012
$ sh -c 'for((i=0;i<3;i++)); do echo -n $i; done; echo'
012
$ ./sh -c 'for((i=0;i<3;i++)); do echo -n $i; done; echo'
./sh: syntax error at line 1: `(' unexpected
$ ./v7sh  -c 'for((i=0;i<3;i++)); do echo -n $i; done; echo'
./v7sh: syntax error at line 1: `(' unexpected
$ dash  -c 'for((i=0;i<3;i++)); do echo -n $i; done; echo'
dash: Syntax error: Bad for loop variable

No problem at all, it accepts and runs it happily and the result is correct (or incorrect?).

Lets see the memory consumptions:

  PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
19166 livibett  20   0 19492  2024  1552 S  0.0  0.1  0:00.03  |   |   `- bash
18890 livibett  20   0 19500  1800  1376 S  0.0  0.1  0:00.00  |   |   `- sh
18838 livibett  20   0  5632   612   508 S  0.0  0.0  0:00.00  |       `- ./sh
26010 livibett  20   0  3980   564   476 S  0.0  0.0  0:00.00  |   |   `- dash
  726 livibett  20   0  3776   400   312 S  0.0  0.0  0:00.00  |   |   `- ./v7sh

You really save not much when running Bash as sh.

File size:

884984 2010-10-10 07:34 /bin/bash
105592 2010-11-10 21:48 /bin/dash
137583 2010-11-10 21:03 sh # Heirloom
 37112 2010-11-10 22:19 sh # V7

V7 shell is creepily small and Bash is a giant.

Note: I didnt touch compiling process for Heirloom, V7. The rest were compiled using -march=core2 -O2 -pipe -fomit-frame-pointer with emerge.

Would I install Dash, Heirloom shell, or V7 shell as my /bin/sh? The answer is no.

pee is a C code from moreutils. You can do something with it like:

% pee bc bc
1+2
1+2
[Press Ctrl+D]
3
3

$ echo 1+2 | pee bc bc
3
3

pee accepts arguments as commands. It will feed those commands with the standard input it gets.

As usual for moreutils post, I wrote a Bash script, pees:

#!/bin/bash

if (($# == 0)); then
  echo "Please specify command(s) as argument(s) of $(basename "$0")."
  exit 1
fi

TMPFILE="$(mktemp)"

while read line; do echo "$line" >> "$TMPFILE"; done

for cmdargs in "$@"; do bash -c "$cmdargs" < "$TMPFILE"; done

rm "$TMPFILE"

Needless to say, pees can do the examples above. But it can also do these as pee can:

ts is a Perl script from moreutils. You use it with pipe, it receives stdout of another program. The basic idea is doing:

command | ts

Its useful when you need to know when a program processes something. But its not enough for me, I wrote a Bash script ts.sh to do more.

#!/bin/bash

DATECMD='date +%H:%M:%S'

process_stdout() {
  while read line; do
    echo -e "\e[44m$($DATECMD)\e[0m $line"
  done
}

process_stderr() {
  while read line; do
    echo -e "\e[41m$($DATECMD)\e[0m $line" 1>&2
  done
}

if (( $# == 0 )); then
  process_stdout
else
  exec 3>&1
  ( bash -c "$*" | process_stdout ) 2>&1 1>&3 | process_stderr
  exec 3>&-
fi

It can timestamp stdout and stderr of a command with different color. See the following example:

http://farm2.static.flickr.com/1380/5160212248_15b5bf58b7.jpg

The timestamps on red is the stderr of emerge, the rest on blue is the stdout. This script could also be used with pipe, but it would only timestamp on stdin, which is a commands stdout via pipe. Its best to use it like:

ts.sh command arg1 arg2 ...

An example with shell script as command:

ts.sh while true \; do echo tik tok \; sleep 1 \; done

chronic is a Perl script from moreutils collection, runs a command quietly. It eats up the output of the command, but if the command exits with error, which is exit status does not equal to zero, then chronic would show you the output.

From its manpage, it provides a clear example how it could be useful with cron:

0 1 * * * chronic backup # instead of backup >/dev/null 2>&1

Note

I use vixin cron, I dont need to redirect stderr to stdout, just chronic backup >/dev/null would do in the old way.

The old way, using redirection is fine, would not give you message sent to stdout since they are redirected to /dev/null, thought you would see get stderr message from log or email if you have sent up.

chronic would get you both.

I made a simple Bash script, chronic.sh:

Sometimes, your script may need to accept a command and run that command in your script. I decided to take a look at $* and $. From manpage:

Special Parameters

* Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, it expands to a single word with the value of each parameter separated by the first character of the IFS special variable. That is, "$*" is equivalent to "$1c$2c...", where c is the first character of the value of the IFS variable. If IFS is unset, the parameters are separated by spaces. If IFS is null, the parameters are joined without intervening separators.

Expands to the positional parameters, starting from one. When the expansion occurs within double quotes, each parameter expands to a separate word. That is, "$" is equivalent to "$1" "$2" ... If the double-quoted expansion occurs within a word, the expansion of the first parameter is joined with the beginning part of the original word, and the expansion of the last parameter is joined with the last part of the original word. When there are no positional parameters, "$" and $ expand to nothing (i.e., they are removed).

I wrote a simple script to understand better:

#!/bin/bash

echo "\$# = $# arguments"
echo

TMP=($*) ; echo -n "(\$*)   => ${#TMP[]} elements => "
for arg in ${TMP[]}; do echo -n "$arg, " ; done ; echo # or in $*

TMP=("$*") ; echo -n "(\"\$*\") => ${#TMP[]} elements => "
for arg in "${TMP[]}"; do echo -n "$arg, " ; done ; echo # or in "$*"

TMP=($) ; echo -n "(\$)   => ${#TMP[]} elements => "
for arg in ${TMP[]} ; do echo -n "$arg, " ; done ; echo # or in $

TMP=("$") ; echo -n "(\"\$\") => ${#TMP[]} elements => "
for arg in "${TMP[]}" ; do echo -n "$arg, " ; done ; echo # or in "$"

echo
echo 'Running:'
"$"
% ./test.sh echo abc '123 456' def
$# = 4 arguments

($*)   => 5 elements => echo, abc, 123, 456, def,
("$*") => 1 elements => echo abc 123 456 def,
($)   => 5 elements => echo, abc, 123, 456, def,
("$") => 4 elements => echo, abc, 123 456, def,

Running:
abc 123 456 def

"$" is the one to use.

If I need to run a short shell script:

% ./test.sh bash -c 'for((i=0;i<3;i++)); do echo -n "$i," ; done ; echo'
$# = 3 arguments

($*)   => 11 elements => bash, -c, for((i=0;i<3;i++));, do, echo, -n, "$i,", ;, done, ;, echo,
("$*") => 1 elements => bash -c for((i=0;i<3;i++)); do echo -n "$i," ; done ; echo,
($)   => 11 elements => bash, -c, for((i=0;i<3;i++));, do, echo, -n, "$i,", ;, done, ;, echo,
("$") => 3 elements => bash, -c, for((i=0;i<3;i++)); do echo -n "$i," ; done ; echo,

Running:
0,1,2,

Or using bash -c "$*":

% ./test.sh echo abc '123 456' def
% ./test.sh 'for((i=0;i<3;i++)); do echo -n "$i," ; done ; echo'

https://i.ytimg.com/vi/pLb8AfLbkPo/mqdefault.jpg

Cat loves sponge as you can see in this video, but if its soaked with water, a sponge bathing? They would run away next time they see you hiding a sponge behind you. This video provides a nice instructions, if you want to try this one-time only bathing technique:

  1. Acquire cat and remove collar.

    First step is always the hardest, they have sixth sense, they can smell your intentions!

  2. Lightly Soap with dampened cloth (warm diluted solution of cat shampoo is best), ~ praise cat throughout ~

    I doubt they really buy it.

  3. Thoroughly rinse with warm damp cloth, then

  4. Gently towel dry. (do not iron)

    I bet some owners would want to try if ironing is more efficient.

I found its always fun to watch cat getting bathed, though there are some cat breeds do like water. And

Wait a minute!

Just finished this hours-project. I happened to read this thread and know the FIGlet and TOIlet, hours later, I am showing you MPDisp. I used FIGlet not TOIlet because TOIlet doesn't support center aligning. Here is a screenshot:

MPDisp

You can read and download the code. The only dependency should be FIGlet (do I really need to say you need MPD?). It doesn't rely on (nc)mpc(pp), it contacts MPD directly. All it does is show the song information and give you a little bit of control, see the key list:

  • p - Pause/Resume
  • Enter - Play
  • s - Stop
  • n - Next song
  • p - Previous song
  • r - Repeat mode on/off
  • S - Single mode on/off
  • R - Random mode on/off
  • q - Quit
  • Q - Quit with MPD
I was thinking to use TOIlet, it has a feature called filter, which can render with colors. But it can't align text center. I might try to see what I can do to next.

The only problem I have seen is if you have too small window, text will be messed up, and I am not intended to solve it right now.