Showing posts with label rwb. Show all posts
Showing posts with label rwb. Show all posts

Wednesday, January 16, 2008

An RWB Primer (from 2005)

vote this up on DZone

Since RWB had been relegated to the dust bin of depracation before I decided to pick it back up and start afresh, I thought it would be worthwhile to post some introductory information about it. This is cribbed from things I wrote at the time of the 0.1.1 release.

Introduction

RWB was written to scratch an itch that ab couldn’t quite reach. I wanted to be able to build a weighted list of urls with which to test a website. RWB will become a little language in which you can write such lists, run tests, and build reports. For now, I’m trying to get the engine and the reporting right. Please use RWB and let me know what else it ought to do. Criticism, suggestions, and especially patches are welcome.

Samples

Here are two quick examples of scripts using RWB and their output. (Note that I’m showing both stdout and stderr in the output.) Here’s a very simple test to start with:

require 'rwb'

urls = RWB::Builder.new()

urls.add_url(10, "http://www.example.com")
urls.add_url(10, "http://www.example.com/nonesuch")
urls.add_url(70, "http://www.example.com/entries")

queries = ['foo+bar', 'bar+baz', 'quux']
urls.add_url_group(10, "http://www.example.com/search?", queries)

tests = RWB::Runner.new(urls, 100, 5)

tests.run
tests.report_header
tests.report_overall([0.5, 0.9, 0.99, 0.999])

This script does a couple of things. It sets up an RWB::Builder object called urls, which contains three Url objects and a UrlGroup object. The three Urls consist of a weight and a URL—the total of the weights don’t matter, they’re not percentages. The UrlGroup has a weight, a base URL and an array of extensions to be added to that URL.

Once the urls object is set, an RWB::Runner object called tests is built. This object is then used to run the tests and build a report header and an overall performance report. The output of running this script is shown below:

$ ruby vsim.rb
completed 10 runs
completed 20 runs
completed 30 runs
completed 40 runs
completed 50 runs
completed 60 runs
completed 70 runs
completed 80 runs
completed 90 runs
completed 100 runs
Concurrency Level:       5
Total Requests:          100
Total time for testing:  1.29918 secs
Requests per second:     76.9716282578242
Mean time per request:   64  msecs
Standard deviation:      9
Overall results:
        Shortest time:  10 msecs
        50.0%ile time:  62 msecs
        90.0%ile time:  78 msecs
        99.0%ile time:  88 msecs
        99.9%ile time:  88 msecs
        Longest time:   88 msecs

Here’s a more involved test, which includes a warmup, using a url_group, and some fancier reporting:

require 'rwb'

urls = RWB::Builder.new()

urls.add_url(2, "http://www.example.com")
urls.add_url(3, "http://www.example.com/nonesuch")
urls.add_url(3, "http://www.example.com/entries")

queries = ['foo+bar', 'bar+baz', 'quux']
urls.add_url_group(10, "http://www.example.com/search?", queries)

tests = RWB::Runner.new(urls, 10_000, 50)

tests.sla_levels = [0.5, 0.9]

tests.warm_up(10)

tests.run
tests.report_header
tests.report_by_time
tests.report_urls([0.5, 0.9, 0.99, 0.999])

This generates the following output:

$  ruby minv.rb
warming up with 10 runs
1 2 3 4 5 6 7 8 9 10
completed 1000 runs
completed 2000 runs
completed 3000 runs
completed 4000 runs
completed 5000 runs
completed 6000 runs
completed 7000 runs
completed 8000 runs
completed 9000 runs
completed 10000 runs
Concurrency Level:       50
Total Requests:          10000
Total time for testing:  38.848752 secs
Requests per second:     257.408526276468
Mean time per request:   41  msecs
Standard deviation:      12
Results by time:
results for requests 0 - 2000
       Shortest time:  23 msecs
       50.0%ile time:  39 msecs
       90.0%ile time:  50 msecs
       Longest time:   93 msecs
results for requests 2000 - 4000
       Shortest time:  24 msecs
       50.0%ile time:  42 msecs
       90.0%ile time:  54 msecs
       Longest time:   68 msecs
results for requests 4000 - 6000
       Shortest time:  25 msecs
       50.0%ile time:  41 msecs
       90.0%ile time:  57 msecs
       Longest time:   82 msecs
results for requests 6000 - 8000
       Shortest time:  26 msecs
       50.0%ile time:  38 msecs
       90.0%ile time:  58 msecs
       Longest time:   81 msecs
results for requests 8000 - 10000
       Shortest time:  25 msecs
       50.0%ile time:  33 msecs
       90.0%ile time:  62 msecs
       Longest time:   102 msecs
Results for http://www.example.com :
       Shortest time:  35 msecs
       50.0%ile time:  52 msecs
       90.0%ile time:  67 msecs
       99.0%ile time:  81 msecs
       99.9%ile time:  98 msecs
       Longest time:   102 msecs
Results for http://www.example.com/nonesuch :
       Shortest time:  24 msecs
       50.0%ile time:  34 msecs
       90.0%ile time:  55 msecs
       99.0%ile time:  62 msecs
       99.9%ile time:  80 msecs
       Longest time:   83 msecs
Results for http://www.example.com/entries :
       Shortest time:  25 msecs
       50.0%ile time:  33 msecs
       90.0%ile time:  55 msecs
       99.0%ile time:  61 msecs
       99.9%ile time:  72 msecs
       Longest time:   79 msecs
Results for http://www.example.com/search?:
       Shortest time:  23 msecs
       50.0%ile time:  34 msecs
       90.0%ile time:  54 msecs
       99.0%ile time:  62 msecs
       99.9%ile time:  88 msecs
       Longest time:   93 msecs

If you’d like to grab rwb and give it a try, it’s available as a gem.

RWB Reborn

With Zed’s recent departure from the Ruby community, I’m going to un-depracate RWB (the Ruby Web Bench). I’d originally stopped development on it for two reasons:

  1. I wasn’t happy with the basic design of the system. I still have concerns about its scalability and its extensibility, but I think both of these are fixable with some redesign/rewriting.
  2. Zed’s rfuzz seemed poised to provide a better solution, but this never really materialized and doesn’t look like it ever will.

I still think there’s a need for a good web benchmarking and testing tool. Something that’s capable of checking multiple URLs, fuzz testing, and providing detailed reports about test results. RWB could be the basis for such a tool.

Before I can do anything else, I need to look into some basic design decisions.

  • How should I assemble, send, receive, and record HTTP requests and responses?
  • How can I better parallelize the work when a single process/system isn’t capable of generating the needed load?
  • What data do I need to keep, and how do I best make it accessible to users wanting to generate their own reports?
  • Do I need to maintain test information across multiple runs?

The answers to these questions will probably force some additional questioning:

  • Is Ruby the right language for all/part of this, or should I be looking to Ragel, C, Erlang, etc.?
  • Should I be providing a single package, or a library and an app that uses it?

So, what does this mean to you? First, if you’re interested in RWB (or are currently using it), I’d like to hear from you. What’s good, bad, and ugly about the current system? Second, if you’re interested in a new and improved website testing/benchmarking tool, what are you looking for? Third, if you’re interested in working on a system like RWB, get in touch with me because I’d love to build a team to do this right.

What am I going to do next? I’m off to read up on httperf and rfuzz—we’ll see where that takes me. Oh, and I've got a couple of pages of planning material I wrote before I depracated RWB for rfuzz -- I guess I should give those a good read too.