As of 2016-02-26, there will be no more posts for this blog. s/blog/pba/
Showing posts with label benchmark. Show all posts

A couple of years back I wrote about the builtin vs keyword conditional expressions, but I didnt think about another conditional use with arithmetic, that is with tests like [ $a -gt $b] or [[ $a -ne 0 ]].

Use similar code to benchmark 1 > 0 as shown below, not the real code, but you get the idea:

time for ((i = 0; i < 10000; i++)); do <test> 1 >/-gt 0 ; done

The result is:

<test> with time % slower
[ 6.647s 47.9%
[[ 4.692s 04.4%
(( 4.493s fastest
test (builtin) 6.538s 45.5%

(( is just marginally faster than [[ by my definition. Of course, they both are faster than builtin [ and test, as for /usr/bin/test, its an external command, there is no point to test, because its slow for sure.

With this result, there really isnt much difference between two, however, 1 > 0 is more readable than 1 -gt 0, literally and mathematically.

Note

Yes, [[ 1 > 0 ]] is a valid syntax, but it doesnt do what you think, its not arithmetically but lexicographically. In short, its for strings, see bash(1), and there is no such this as [[ 1 >= 0 ]].

Even since I started writing Bash script, I always use [[ for if conditional statement, the double square brackets, never the single one version unless compatibility has to be taken into account. Why did I choose it? Its because I read [[ is faster. Ive never tried to confirm it by myself, although I do know that [ is a type of shell builtin command and [[ is type of shell keyword, and [ is equivalent to test, which is also a builtin command in Bash for very long time.1

Note

[ and [[ can also be used for arithmetic comparison, such as the use of -gt operator, see another test with Arithmetic Evaluation, ((. This post only focuses on the strings.

From time to time, I often see people still using [ even in the script with a few Bash-only syntaxes or features. They are clearly not writing with compatibility as a requirement. If I have a chance, then I would probably advise the coder to change to [[. I had done so a few times in the past.

However, I never see the numbers, so I used the following to test:

time for ((i = 0; i < 10000; i++)); do <test> -z '' ; done

The result is:

<test> with time % slower
[ 00.149s 00,063.7%
[[ 00.091s fastest
test (builtin) 00.150s 00,064.8%
/usr/bin/test 13.798s 15,063.6%

[ and test possibly are synonyms since they are pretty close.

As you can see [[ definitely is the winner, and /usr/bin/test external command is the slowest. The problem with /usr/bin/test isnt that is inefficient, but external command is costly.

If you are new to Bash, just use [[.

[1]If you dont know the differences between keyword, builtin command, and external command, google them.

1   Firefox 4 and Opera

Just heard3 about this new benchmark Kraken4, so I tried it with FF4 beta and Opera 10.61. I only have two browsers on my system currently, its too bad that I dont have Chromium to compare because the results are quite interesting

TEST                         COMPARISON            FROM                 TO               DETAILS
                                                  (Opera 10.61)   (FF 4.0b7pre 20100914)
====================================================================================

** TOTAL **:                 1.50x as fast     19627.2ms +/- 2.3%   13061.5ms +/- 1.1%     significant

====================================================================================

  ai:                        1.41x as fast      3344.5ms +/- 13.0%    2371.8ms +/- 6.4%     significant
    astar:                   1.41x as fast      3344.5ms +/- 13.0%    2371.8ms +/- 6.4%     significant

  audio:                     1.55x as fast      6657.7ms +/- 1.2%    4307.2ms +/- 1.0%     significant
    beat-detection:          1.071x as fast     1301.8ms +/- 3.7%    1215.3ms +/- 2.0%     significant
    dft:                     2.54x as fast      2844.6ms +/- 2.8%    1119.4ms +/- 2.7%     significant
    fft:                     1.051x as fast     1137.7ms +/- 1.3%    1082.4ms +/- 3.0%     significant
    oscillator:              1.54x as fast      1373.6ms +/- 1.8%     890.1ms +/- 0.9%     significant

  imaging:                   2.30x as fast      7628.8ms +/- 1.4%    3318.3ms +/- 1.6%     significant
    gaussian-blur:           3.46x as fast      5256.8ms +/- 1.8%    1517.7ms +/- 0.8%     significant
    darkroom:                1.67x as fast       989.6ms +/- 1.9%     592.7ms +/- 0.8%     significant
    desaturate:              1.144x as fast     1382.4ms +/- 1.3%    1207.9ms +/- 3.8%     significant

  json:                      *1.32x as slow*     302.4ms +/- 1.4%     399.1ms +/- 0.7%     significant
    parse-financial:         *1.88x as slow*     134.9ms +/- 1.9%     253.9ms +/- 0.8%     significant
    stringify-tinderbox:     1.154x as fast      167.5ms +/- 2.1%     145.2ms +/- 1.0%     significant

  stanford:                  *1.57x as slow*    1693.8ms +/- 2.8%    2665.1ms +/- 0.6%     significant
    crypto-aes:              *1.93x as slow*     377.8ms +/- 9.8%     729.6ms +/- 0.6%     significant
    crypto-ccm:              *1.109x as slow*    473.2ms +/- 10.5%     524.7ms +/- 1.9%     significant
    crypto-pbkdf2:           *1.85x as slow*     634.3ms +/- 4.0%    1176.0ms +/- 0.7%     significant
    crypto-sha256-iterative: *1.126x as slow*    208.5ms +/- 1.1%     234.8ms +/- 1.6%     significant

As you can see FF4 is faster in three categories of tests: ai, audio and imaging; and slower in json and crypto categories. Last month, from results of SunSpider, Opera 10.61 is faster than FF4.0b5pre. Now, with tests above, FF4.0b7pre is faster.

[3]The original link was http://blog.mozilla.com/rob-sayre/2010/09/14/release-the-kraken/, but it returns 410 GONE.
[4]The original link was http://www.krakenbenchmark.com/, the content was gone.

2   Firefox 3.6.9

===============================================
RESULTS (means and 95% confidence intervals)
-----------------------------------------------
Total:                       27035.3ms +/- 1.2%
-----------------------------------------------

  ai:                         4529.0ms +/- 5.3%
    astar:                    4529.0ms +/- 5.3%

  audio:                      9842.9ms +/- 1.3%
    beat-detection:           2299.4ms +/- 2.2%
    dft:                      3488.6ms +/- 1.8%
    fft:                      2212.1ms +/- 3.0%
    oscillator:               1842.8ms +/- 4.0%

  imaging:                    7502.1ms +/- 1.5%
    gaussian-blur:            3481.2ms +/- 2.8%
    darkroom:                  834.8ms +/- 0.8%
    desaturate:               3186.1ms +/- 2.5%

  json:                        520.9ms +/- 1.2%
    parse-financial:           350.7ms +/- 1.4%
    stringify-tinderbox:       170.2ms +/- 1.9%

  stanford:                   4640.4ms +/- 1.0%
    crypto-aes:               1367.0ms +/- 0.9%
    crypto-ccm:               1028.7ms +/- 1.4%
    crypto-pbkdf2:            1676.7ms +/- 1.8%
    crypto-sha256-iterative:   568.0ms +/- 0.6%

I ran it four times, two of them crashed Firefox and this test used a lot of memory, more than 1 GB. I am also compiling Chromium for this benchmark, result will be added later.

3   Firefox ESR 17.0.2 with Kraken 1.1

===============================================
RESULTS (means and 95% confidence intervals)
-----------------------------------------------
Total:                        6688.6ms +/- 1.4%
-----------------------------------------------

  ai:                          236.7ms +/- 3.7%
    astar:                     236.7ms +/- 3.7%

  audio:                      2528.1ms +/- 3.7%
    beat-detection:            610.8ms +/- 1.1%
    dft:                      1047.5ms +/- 6.1%
    fft:                       444.9ms +/- 0.6%
    oscillator:                424.9ms +/- 18.1%

  imaging:                    2505.4ms +/- 2.4%
    gaussian-blur:            1438.1ms +/- 4.1%
    darkroom:                  550.3ms +/- 0.2%
    desaturate:                517.0ms +/- 4.3%

  json:                        276.8ms +/- 2.7%
    parse-financial:           152.6ms +/- 3.6%
    stringify-tinderbox:       124.2ms +/- 4.4%

  stanford:                   1141.6ms +/- 1.4%
    crypto-aes:                263.0ms +/- 1.5%
    crypto-ccm:                199.5ms +/- 1.5%
    crypto-pbkdf2:             509.0ms +/- 1.7%
    crypto-sha256-iterative:   170.1ms +/- 3.7%

4   Chromium 7.0.517.5

===============================================
RESULTS (means and 95% confidence intervals)
-----------------------------------------------
Total:                        22962.1ms +/- 0.6%
-----------------------------------------------

  ai:                          1220.5ms +/- 0.5%
    astar:                     1220.5ms +/- 0.5%

  audio:                       8739.2ms +/- 0.8%
    beat-detection:            2288.9ms +/- 1.1%
    dft:                       3169.9ms +/- 2.1%
    fft:                       2370.6ms +/- 0.5%
    oscillator:                 909.8ms +/- 0.6%

  imaging:                    11067.4ms +/- 0.9%
    gaussian-blur:             5549.8ms +/- 1.9%
    darkroom:                  2748.7ms +/- 1.7%
    desaturate:                2768.9ms +/- 1.1%

  json:                         890.2ms +/- 0.3%
    parse-financial:            507.0ms +/- 0.4%
    stringify-tinderbox:        383.2ms +/- 0.4%

  stanford:                    1044.8ms +/- 0.7%
    crypto-aes:                 229.3ms +/- 0.9%
    crypto-ccm:                 190.3ms +/- 0.5%
    crypto-pbkdf2:              435.0ms +/- 0.8%
    crypto-sha256-iterative:    190.2ms +/- 1.4%

5   Summary

http://chart.apis.google.com/chart?chxr=0,0,30000&chxt=y&chbh=a&chs=628x240&cht=bvg&chco=FF9900,80C65A,76A4FB,FFCC33&chds=0,30000,0,30000,0,30000,0,30000&chd=t:13061.5|19627.2|22962.1|27035.3&chdl=Firefox+4|Opera+10|Chromium+7|Firefox+3.6&chdlp=b&chma=0,0,0,10|0,35
Browser   Version     Total Time   To FF4
Firefox    4.0b7pre   13061.5 ms  --------
Opera     10.61       19627.2 ms  + 50.27%
Chromium   7.0.517.5  22962.1 ms  + 75.80%
Firefox    3.6.9      27035.3 ms  +106.98%