As of 2016-02-26, there will be no more posts for this blog. s/blog/pba/
Showing posts with label memory. Show all posts

I came across Glances, a system monitoring program written in Python, as its name hints, it enables you to review system information, status, or resource usage in just a few glances. As you can see from the screen shot above, all information in one screen.

  • CPU: Glances probably the only few programs which display the details of what CPU spends time on rather than just a simple percentage of utilization. How much time it's idling or waiting for IO? For example, iowait is an useful information, when disk is busy, more or less it could slow down the system for different reasons and by reading iowait, you know some heavy IO activity is going on.
  • Load average: It's same as uptime gives, the load average of past 1, 5, and 15 minutes.
  • Memory and Swap: Like CPU, more detail is shown, such as buffered, cached, active, or inactive memory. Similar to free command output.
  • Network: Each interface's bandwidth usage, upload and download rates. You can press B to switch unit between bit/second and byte/second.
  • Disk I/O: Read/Write rates on partitions and devices. Not only physical storage devices but also optical devices.
  • Mount points: Similar information you get from df command, such as total size, used space, and available size.
  • Processes: Like htop or top, you can use keys to choose sorting fields.
Glances shows information as much as they can be fitted in terminal window size. You can turn off sections by keys, separately, if you don't need them.

It also support server mode, a client can use XML-RPC call to get the system information in JSON format. It's possible to write your own client if you don't like default text-based client or you can write a GUI or web interface to display the data. It supports Linux, Mac OS, and Windows, although no default client for Windows, it will only runs as server due to no curses module available on Windows platform.

Glances is only one-year-old, first released on December 4, 2011, there are many possibilities for improvements. For example, configuration file for colors or disabling sections by default. Also custom fields for processes, and the list can go on and on.

I was intrigued by its name when I first heard of it and the idea behind this program is useful and simple. You don't need to run several programs to get all information of a system. You can have everything on one screen, even status of multiple systems at same time with client/server mode and terminal multiplexer.

I wanted to see how much memory Firefox 4 use, not for any particular reason. So, I quickly wrote one short script:

#ff_pid=1234
max_mem=2048
max_bar_len=100
while :; do
ff_pid=$(ps -e -o pid,cmd | grep firefox-bin$ | grep -o '^ *[0-9]*')
if [[ $ff_pid ]]; then
  mem_mb=$(($(ps -p $ff_pid -o rss=)/1024))
  bar_len=$((max_bar_len*mem_mb/max_mem))
  printf "$(date +%H:%M) %4dM [%3d%%] \033[41m%${bar_len}s\033[42m%$((max_bar_len-bar_len))s\033[0m\n" $mem_mb $((100*mem_mb/max_mem)) '' ''
else
  printf "%s ---\n" $(date +%H:%M)
fi
sleep 60
done | tee ff_mem.txt | awk '{if(m!=$2){print $0;m=$2}}'

max_mem specifies the system memory in MB. max_bar_len is the how long the bar can be in character if memory use is 100%. Because I always run additional Firefox instance (profile, using --no-remote), so I have to make sure I get the main Firefoxs PID, which is the one I want to monitor. The awk is used for removing consecutive lines of same amount memory use. You can take it out if you want to see minute by minute results.

Here is a result after I started up Firefox today:

http://farm6.static.flickr.com/5081/5353295127_55ef2c1e6f_b.jpg

I havent fully used Firefox to visit sites I often do. Normally, after it reaches about 20%, I restart Firefox. I want to see if it will go down after it reaches 50%.

Nope, this is not about the alphabet song at all. Its about how fast a Python code can generate 'abcdefghijklmnopqrstuvwxyz'.

The answer is already there, 'abcdefghijklmnopqrstuvwxyz'. Still dont see it? Alright, let me give you a full Python code:

#!/usr/bin/env python
# A program demonstrates how to generate an 'abcdefghijklmnopqrstuvwxyz' QUICK.


atoz = 'abcdefghijklmnopqrstuvwxyz'

Thats the complete code, get it?

You might ask Seriously, you are writing a post about this? Yes, Im.

I found many people have been trying to generate such string in different ways, more Pythonic, looks like smart or genius, or looks like hell. But sometimes, the direct approach is the most simple way and easy way to understand, not only by the programmer but also by people who dont know about programming. The following list is what I saw from Internet, there must be more.

atoz = 'abcdefghijklmnopqrstuvwxyz'
from string import ascii_lowercase
atoz = map(chr, xrange(97, 123))
atoz = map(chr, xrange(ord('a'), ord('z') + 1))
atoz = map(chr, range(97, 123))
atoz = [chr(i) for i in xrange(97, 123)]

Which one is the best in your mind?

1   Runtime Profiling

I did some profiling, yep, I did. Here is the code for profiling:

#!/usr/bin/env python

import sys
import timeit

def main():

  m = [
      ("atoz = 'abcdefghijklmnopqrstuvwxyz'", '', 0),
      ("from string import ascii_lowercase", '', 0),
      ("atoz = map(chr, xrange(97, 123))", '', 2),
      ("atoz = map(chr, xrange(ord('a'), ord('z') + 1))", '', 2),
      ("atoz = list(map(chr, range(97, 123)))", '', 3),
      ("atoz = list(map(chr, range(ord('a'), ord('z') + 1)))", '', 3),
      ("atoz = [chr(i) for i in xrange(97, 123)]", '', 2),
      ("atoz = [chr(i) for i in range(97, 123)]", '', 3),
      ('atoz[13]', "atoz = 'abcdefghijklmnopqrstuvwxyz'", 0),
      ('atoz[13]', "atoz = map(chr, xrange(97, 123))", 2),
      ('atoz[13]', "atoz = list(map(chr, range(97, 123)))", 3),
      ]
  max_len = max(map(lambda x: len(x[0]), m))

  for i in range(len(m)):
    stat, setup, v = m[i]
    if v != 0 and v != sys.version_info[0]:
      continue
    if setup:
      sys.stdout.write('%s\n' % setup)
    else:
      setup = 'pass'
    sys.stdout.write(('%%-%ds -> ' % max_len) % stat)
    sys.stdout.write('%12.6f us\n\n' % (min(timeit.Timer(stat, setup).repeat(10000, 100)) * 1000000))

if __name__ == '__main__':
  main()

At first, I used cProfile, but it only shows to milliseconds level, so I switched to timeit. Each one is ran for 10,000 sessions and each session accumulate 10 runs of the code, then pick up the smallest runtime as result. Which represent how fast the code might be able to be executed. But there is a catch, it might actually be run faster than that just timeit could not tell because timers precision.

Here is the results:

CODE                                                       Python 2.5.4    Python 2.6.5    Python 3.1.2
atoz = 'abcdefghijklmnopqrstuvwxyz'                  ->     4.768372 us     4.768372 us     3.814697 us

from string import ascii_lowercase                   ->   186.920166 us   177.860260 us   258.922577 us

atoz = map(chr, xrange(97, 123))                     ->   584.840775 us   560.998917 us   vvvvvvvvvvvvv
atoz = map(chr, range(97, 123))                      ->   618.934631 us   577.926636 us   vvvvvvvvvvvvv
atoz = list(map(chr, range(97, 123)))                ->   ^^^^^^^^^^^^^   ^^^^^^^^^^^^^   775.814056 us

atoz = map(chr, xrange(ord('a'), ord('z') + 1))      ->   609.874725 us   594.854355 us   vvvvvvvvvvvvv
atoz = list(map(chr, range(ord('a'), ord('z') + 1))) ->   ^^^^^^^^^^^^^   ^^^^^^^^^^^^^   811.100006 us

atoz = [chr(i) for i in xrange(97, 123)]             ->   942.945480 us   875.949860 us   vvvvvvvvvvvvv
atoz = [chr(i) for i in range(97, 123)]              ->   ^^^^^^^^^^^^^   ^^^^^^^^^^^^^   946.044922 us

atoz = 'abcdefghijklmnopqrstuvwxyz'
atoz[13]                                             ->    12.874603 us     9.775162 us     7.867813 us

atoz = map(chr, xrange(97, 123))
atoz[13]                                             ->    10.967255 us     7.867813 us     vvvvvvvvvvv
atoz = list(map(chr, range(97, 123)))
atoz[13]                                             ->    ^^^^^^^^^^^^     ^^^^^^^^^^^     7.867813 us

As you can see, using constant from string module, map, and list comprehensions are very slow. Yes, its one-time setup but the direct approach is also the one-time setup. And those, except the ascii_lowercase, would take everyone sometime to read it and understand.

2   Memory Profiling

I also did a memory profiling using this code with Guppy-PE:

#!/usr/bin/env python

from guppy import hpy

hp = hpy()

hp.setrelheap()
atoz = 'abcdefghijklmnopqrstuvwxyz'
print hp.heap()
print

hp.setrelheap()
from string import ascii_lowercase
print hp.heap()
print

hp.setrelheap()
print hp.heap()
atoz = map(chr, xrange(97, 123))
print

hp.setrelheap()
atoz = map(chr, xrange(ord('a'), ord('z') + 1))
print hp.heap()
print

hp.setrelheap()
atoz = map(chr, range(97, 123))
print hp.heap()
print

hp.setrelheap()
atoz = [chr(i) for i in xrange(97, 123)]
print hp.heap()
print

The results:

=== Python 2.5.4 ===

Partition of a set of 1 object. Total size = 656 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100      656 100       656 100 types.FrameType

Partition of a set of 1 object. Total size = 560 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100      560 100       560 100 types.FrameType

Partition of a set of 1 object. Total size = 656 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100      656 100       656 100 types.FrameType

Partition of a set of 2 objects. Total size = 888 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  50      560  63       560  63 types.FrameType
     1      1  50      328  37       888 100 list

Partition of a set of 2 objects. Total size = 984 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  50      656  67       656  67 types.FrameType
     1      1  50      328  33       984 100 list

Partition of a set of 2 objects. Total size = 888 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  50      560  63       560  63 types.FrameType
     1      1  50      328  37       888 100 list

=== Python 2.6.5 ===

Partition of a set of 1 object. Total size = 448 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100      448 100       448 100 types.FrameType

Partition of a set of 1 object. Total size = 448 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100      448 100       448 100 types.FrameType

Partition of a set of 1 object. Total size = 448 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1 100      448 100       448 100 types.FrameType

Partition of a set of 2 objects. Total size = 776 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  50      448  58       448  58 types.FrameType
     1      1  50      328  42       776 100 list

Partition of a set of 2 objects. Total size = 776 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  50      448  58       448  58 types.FrameType
     1      1  50      328  42       776 100 list

Partition of a set of 2 objects. Total size = 776 bytes.
 Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
     0      1  50      448  58       448  58 types.FrameType
     1      1  50      328  42       776 100 list

Guppy isnt compatible with Python 3, hence there is no result of Python 3.1.2.

Those use map or list comprehension would result a list and required more memory as expected, it doesnt really affect how you use it. Usually you just access it like atoz[10], it works for list and string types. But memory use tells you that string type uses less memory, however, if you notice the runtime result above, you would have seen accessing list element is faster than substring of a string.

3   Conclusion

My conclusion is wgasa.

When you are using big program like Firefox yes, it's big, some of memory usages may go into swap. However, if your harddisk's performance is like mine, it's totally in pain. One word only can describe slow.

There is a way to move the content in swap back to main memory, run the following as root:
swapoff -a
swapon -a

Doesn't look genius to me, but I don't know if there is any way else. If it is, please leave a comment.

You may also want to change the system setting, so reduce the chance of moving back. On my Fedora 10, the default vm.swappiness is 60. You can change it on-the-fly as root:
sysctl vm.swappiness=10
And edit the /etc/sysctl.conf as root:
vm.swappiness = 10

If you don't want to use swap anymore, after you swapoff, then to remove the swap partition by removing from /etc/fstab and fdisk'd.