measurements

Run your own Ruby Code Interrogation

jonforums
3
1
Ruby

What?

A mini-framework for hosting your own little Ruby Code Interrogation party.
Currently benchmarking, API tracing, and controlled execution for use with
code profiling is envisioned.

The project’s goal is to keep as much of the boilerplate setup and driver code
needed to automate benchmarking, API tracing, and profiling out of your test
code. RCI provides a set of core workloads, but you’ll likely want to create
custom workloads for your needs. Either create a new core workload by dropping
a .rb workload file into the workloads subdirectory, or create custom workloads
by:

  1. creating a directory for your .rb workload files
  2. setting the directory from (1) as the value of :my_workloads in config.rb
  3. writing workloads in .rb files and drop them into the directory from (1)

Once you’ve got your workloads and config.rb configuration correct, use the
rci runner to interrogate your code.

Custom plugins are created by dropping an .rb file (starting with require 'rci')
into the plugins subdirectory. The plugin can count on all framework
configuration being complete before the plugin is loaded, and the plugin is
loaded before any COMMAND is executed.

Note that using the trace or profile commands typically requires you to use
an administrator level shell.

Setup and Configuration

  1. Clone the GitHub repo
    (or extract a zipball)
    via git clone git://github.com/jonforums/measurements.git
  2. Change to the directory from step (1) and edit the config.rb file to your
    liking. If you want to perform API Tracing on Windows via the trace command,
    download Process Monitor,
    extract it to a directory, and enter that absolute directory path into the
    :tracer: section of config.rb, similar to the project’s default. If you
    want to perform profiling on Windows, configure the :profiler: section.
    Currently, only support for VTune AmplifierXE
    has been implemented.
  3. Initialize the framework by running rci init (Windows) or ./rci init (Linux)
  4. That’s it. You’re ready to start interrogating Ruby with the included core
    workloads or custom workloads of your own!

Usage


C:\projects\measurements-git>rci

usage: rci [RUBY_OPTS] COMMAND [CMD_OPTS]

where COMMAND is one of:
  bench <W|all>   benchmark workload W or all workloads
  exec W          execute workload W
  init            initialize environment
  ls              list all workloads
  profile W       profile workload W
  trace W         trace workload W

generic options:
  -h, --help      show this help message
  -v, --version   show RCI version

where RUBY_OPTS are:
  --disable-gems  disable RubyGems use

where common CMD_OPTS are:
  --extended      use extended duration workload

where 'exec' CMD_OPTS are:
  --pause         pause before/after running workload

Current Usage Warts

  • Must manually set the environment variable JRUBY_OPTS=--1.9 to
    interrogate JRuby in 1.9 mode. As of JRuby 1.6.5, you can also use a
    .jrubyrc file to configure JRuby.

Examples

What core workloads are available?

C:\measurements>rci ls

=== Core Workloads ===

 * core_brd_filelines_crlf
 * core_brd_filelines_lf
 * core_pi
 * core_rd_filelines_crlf
 * core_rd_filelines_lf
 * core_require_empty
 * core_require_nested

How do I list all my custom workloads?

Assuming you’ve correctly configured :my_workloads in your config.rb file to
point to the directory containing your .rb workload files, it’s as simple as:

C:\measurements>rci ls

=== Core Workloads ===

 * core_brd_filelines_crlf
 * core_brd_filelines_lf
 * core_pi
 * core_rd_filelines_crlf
 * core_rd_filelines_lf
 * core_require_empty
 * core_require_nested

=== My Workloads ===

 * super_fast_require
 * fast_io_write
 * fast_malloc

How do I benchmark a workload?

C:\measurements>rci bench core_require_missing
ruby 1.9.2p174 (2011-01-28 revision 30696) [i386-mingw32]
Rehearsal --------------------------------------------------------
core_require_missing  10.998000  20.327000  31.325000 ( 31.403797)
---------------------------------------------- total: 31.325000sec

                           user     system      total        real
core_require_missing  11.232000  20.187000  31.419000 ( 31.631809)

How do I benchmark a workload and disable RubyGems?

C:\measurements>rci --disable-gems bench core_pi
ruby 1.9.3dev (2011-02-08 trunk 30825) [i386-mingw32]
RubyGems disabled
Rehearsal ------------------------------------------
core_pi  1.606000   0.328000   1.934000 (  1.939111)
--------------------------------- total: 1.934000sec

             user     system      total        real
core_pi  1.701000   0.202000   1.903000 (  1.923110)

How do I execute a workload and pause to profile it?

C:\measurements>rci exec core_brd_filelines_crlf --pause
Press <ENTER> to start executing workload:

[INFO] executing 'core_brd_filelines_crlf' workload

Press <ENTER> to finish:

How do I use pik to benchmark with multiple Rubies?

C:\measurements>pik run rci --disable-gems bench core_pi

jruby 1.6.0.RC1 (ruby 1.8.7 patchlevel 330) (2011-01-10 769f847)
(Java HotSpot(TM) Client VM 1.6.0_23) [Windows 7-x86-java]
Rehearsal -------------------------------------------
core_pi   2.437000   0.000000   2.437000 (  2.417000)
---------------------------------- total: 2.437000sec

              user     system      total        real
core_pi   2.357000   0.000000   2.357000 (  2.357000)


ruby 1.8.7 (2010-12-23 patchlevel 330) [i386-mingw32]
Rehearsal -------------------------------------------
core_pi   1.778000   0.328000   2.106000 (  2.100120)
---------------------------------- total: 2.106000sec

              user     system      total        real
core_pi   1.857000   0.234000   2.091000 (  2.137122)


ruby 1.9.2p174 (2011-01-28 revision 30696) [i386-mingw32]
RubyGems disabled
Rehearsal -------------------------------------------
core_pi   1.669000   0.156000   1.825000 (  1.879108)
---------------------------------- total: 1.825000sec

              user     system      total        real
core_pi   1.607000   0.234000   1.841000 (  1.865107)


ruby 1.9.3dev (2011-02-02 trunk 30760) [i386-mingw32]
RubyGems disabled
Rehearsal ------------------------------------------
core_pi  1.763000   0.219000   1.982000 (  1.980113)
--------------------------------- total: 1.982000sec

             user     system      total        real
core_pi  1.731000   0.218000   1.949000 (  1.952112)


ruby 1.9.3dev (2011-02-08 trunk 30825) [i386-mingw32]
RubyGems disabled
Rehearsal ------------------------------------------
core_pi  1.701000   0.234000   1.935000 (  1.928110)
--------------------------------- total: 1.935000sec

             user     system      total        real
core_pi  1.747000   0.172000   1.919000 (  1.927110)

How do I use rvm to benchmark multiple Rubies on Linux?

[jon@archee measurements-git]$ rvm system,jruby do ./rci bench core_pi
ruby 2.0.0dev (2011-11-06 trunk 33644) [i686-linux]
Rehearsal -------------------------------------------
core_pi   1.690000   0.060000   1.750000 (  1.741412)
---------------------------------- total: 1.750000sec

              user     system      total        real
core_pi   1.690000   0.050000   1.740000 (  1.738589)

jruby 1.6.5 (ruby-1.8.7-p330) (2011-10-25 9dcd388) (Java HotSpot(TM) Client VM 1.7.0_01) [linux-i386-java]
Rehearsal -------------------------------------------
core_pi   3.027000   0.000000   3.027000 (  2.990000)
---------------------------------- total: 3.027000sec

              user     system      total        real
core_pi   2.883000   0.000000   2.883000 (  2.882000)

How do I easily profile a workload but not use the manual exec --pause method?

Assuming you have a a profiler provider properly configured in
config.rb, you can run something similar to:

C:\measurements>rci --disable-gems profile core_pi
[INFO] running 'AmplifierXE' profile provider

Summary
-------

Elapsed Time:  1.014
CPU Time:      0.685

How do I extend the workload execution time to get more profiling data?

Edit the :extended_iterations value in config.rb and run something
similar to:

C:\measurements>rci --disable-gems profile core_rd_filelines_lf --extended
[INFO] running 'AmplifierXE' profile provider

Summary
-------

Elapsed Time:  407.274
CPU Time:      404.298

License

3-clause BSD. See project LICENSE file.

TODO

  • fix exception on unknown command; implement known commands list
  • add basic specs using minitest/spec
  • fix JRuby problem in processmonitor.rb (IO.popen?) when running trace
  • implement trace (strace and ltrace) on Linux
  • add --provider P to trace and profile commands
  • embed trollop if cmd line parsing becomes too burdensome
  • investigate adding measure command using hitimes gem
  • create rci.ps1
  • extract cmd line arg parsing from rci.bat to C .exe
  • support both system and user workloads and input working dirs
  • implement basic benchmarking statistics
  • implement cmd line regex workload matching for benchmarking