NAME HTTP::LoadGen - a HTTP load generator toolset SYNOPSIS use HTTP::LoadGen; use HTTP::LoadGen qw/:all/; # ask import() to replace the built-in 'rand' operator by our # thread-specific RNG (uses *CORE::GLOBAL::rand) use HTTP::LoadGen qw/-rand :all/; ###################### # the load generator # ###################### HTTP::LoadGen::loadgen \%config; ####################### # auxiliary functions # ####################### # process management # create a collection of worker processes $handle=HTTP::LoadGen::create_proc $nproc, $inithnd, $handler, $exithnd; # start main processing and wait for then to finish %result=%{HTTP::LoadGen::start_proc $handle}; # thread management # create a collection of threads $sem=HTTP::LoadGen::ramp_up $procnr, $nproc, $start, $max, $duration, $handler; # wait for them to finish $sem->down; # idle a bit HTTP::LoadGen::delay $prefix, \%param; # get current thread number $nr=HTTP::LoadGen::threadnr; # get the configuration hash $config=HTTP::LoadGen::options; # get/set thread-specific user data $data=HTTP::LoadGen::userdata; HTTP::LoadGen::userdata=$data; # get/set thread specific random number generator $rng=HTTP::LoadGen::rng; HTTP::LoadGen::rng=$rng; # next random number $random=HTTP::LoadGen::rnd $max; INSTALLATION perl Makefile.PL make make test make install DEPENDENCIES * perl 5.8.8 * IPC::ScoreBoard * Coro * AnyEvent * Async::Interrupt * Net::SSLeay DESCRIPTION This module implements a multi-process and multi-thread load generator for HTTP. It uses Coro threads. So, in reality it does not use threads but event-based IO. Features * limited support for SSL connections * keep-alive connections * configurable delay before and after each request * run a list of URLs many times * compute next URL based on the current request * DNS cache can be preinitialized * slow ramp up * request bodies * custom request headers Overview Note, this POD is best view via Apache2::PodBrowser. Parallelism The load generator follows a 2-level supervisor-worker pattern. The central function, "loadgen", creates a certain number of child processes. Each child process then creates in a slow ramp up phase worker threads up to a configurable total upper thread limit. The thread limit is configured independent on the number of worker processes. You configure a number of processes that is about 1.5-5 times the number of available CPUs. The number of threads can then be say 50 or 500 or even 5996 or so. Processes and threads are numbered starting from 0. So, assuming there are 3 processes and 10 threads configured the following table shows how the threads are spread among the processes: Process | Threads --------+------------ 0 | 0 3 6 9 1 | 1 4 7 2 | 2 5 8 Process 0 will run 4 threads, the other 2 processes 3 threads each. The number of threads per process can be calculated as: $TotalThreadCount / $NProc + ($ProcNr < $TotalThreadCount % $NProc) where $NProc is the number of processes used, $ProcNr the number of the current process and $TotalThreadCount the system-wide thread number. $ProcNr ranges from 0 to "$NProc - 1". At the beginning ot the ramp-up phase each process starts up a certain number of threads (maybe 0) to reach the configured start-up thread number. The configured ramp-up duration then determines in which intervals new threads are added. So assuming the threads run long enough you start up with a certain level of parallelism which increases linearly over a certain time interval up to the configured maximum. The Scoreboard The multi-process model of "HTTP::LoadGen" means that each process knows only about its own threads. Sometimes you may want to log for example the overall number of active requests when a new request is started. Or you may want to increment a shared variable for each request to see the progress of an active load run. HTTP::LoadGen::ScoreBoard or IPC::ScoreBoard may be used to achieve that. The Logger "HTTP::LoadGen" doesn't have logging built-in. Instead HTTP::LoadGen::Logger is provided. Random numbers and repeatable results "loadgen" needs for certain operations random numbers. If you need repeatable results that is you want to repeat the same test with the same delays between requests later then you need the same sequence of random numbers for each thread. But the random number generator built-in to Perl is process-wide. "HTTP::LoadGen" provides an interface to set an RNG per thread. CPAN modules like Math::Random::MT use an object oriented approach. So, it may be a good idea to create such an object for each thread and register it with "HTTP::LoadGen". A "ThreadInit" handler is a good place to do that. If the "import" function of "HTTP::LoadGen" is called with the "-rand" parameter ("use HTTP::LoadGen qw/-rand/") the Perl built-in "rand" operator is overwritten (by means of *CORE::GLOBAL::rand) to use the thread-specific RNG. Though, occurences of "rand" in the code that have been compiled before "HTTP::LoadGen" is loaded continue to use the built-in operator. Phases There are several phases in the lifetime of a load run, a process, a thread or a request that can be hooked. A hook is a code reference. ParentInit and ParentExit these 2 hooks run in the parent process. The "loadgen" function checks the configuration and then calls "ParentInit". "ParentExit" is called just before "loadgen" returns. "ParentInit" can start Coro threads. They will run while the process is waiting for the worker children to finish. One thing to consider to do in a "ParentInit" hook is the creation of a HTTP::LoadGen::ScoreBoard. ProcInit and ProcExit these 2 hooks are called in each worker process. When a worker process is started "ProcInit" is called. But before the actual load generation is started the process waits for a signal from the parent process that is sent when every worker process has finished its "ProcInit" phase. So, even if the "ProcInit" phase takes a bit longer it does not influence the load generation other than it is started a bit later. "ProcInit" can start Coro threads. They will run while the process is waiting for the signal from the parent process to start load generation and of course after that until they finish. One thing that should probably be done in a "ProcInit" handler is reseeding of the random number generator. If you need repeatable results then you need a random number generator per thread. The built-in RNG is no help then. However, there are several object oriented RNGs on CPAN. Use the "rng" function to set a thread-specific RNG and "rnd" instead of the built-in "rand" to call it. Another one is the creation of a logger, see HTTP::LoadGen::Logger. "ProcExit" is called after the load generation is over just before the worker process exists. If a "ProcExit" hook is installed its return value determines the exit code of the worker process. Closing the logger would be good here. ThreadInit and ThreadExit these 2 hooks wrap the load generation phase of each thread. If a thread needs private data the "ThreadInit" handler can create and return it. It is then passed to all other hooks called during the lifetime of the thread. Things to consider to do in a "ThreadInit" handler would include 1. registration of a thread-specific random number generator 2. registration of the thread with the scoreboard InitURLs during the load generation phase each thread fetches a list of URLs several times. The actual list is not given as an array or similar but as an interator generator, that is a function that returns a function that returns an URL to be fetched. The "InitURLs" iterator generator is called each time a thread starts another round of fetching URLs. The iterator itself is then called to get the next URL to be fetched. If it returns "undef" or the empty list the current round is over. Then if the configured number of rounds is reached the thread ends or the next round is started (and the "InitURLs" handler is called again). In most cases this complex URL handling is not necessary. Instead one simply needs to check off all items of a predefined list. For these situations a few predefined iterator generators exist. ReqStart and ReqDone these 2 hooks wrap each HTTP request. Here the request would be accounted with the scoreboard. In "ReqDone" logging would occur. HTTP::LoadGen::loadgen \%data "loadgen" is the central function of this module. It starts up child processes, creates threads, generates the load and waits for that all to finish. It returns when all is done. The %data hash passed by reference configures "loadgen" and describes what to do. "loadgen" copies the hash so that the original hash is not changed but the copying is not recursive. If a hash value is an array and one of the hooks changes it that change will be reflected in the original %data hash. However, if you add new hash elements in a hook function they won't show up in %data after "loadgen" returns. Request descriptor and return element A number of elements of the %data hash are hook functions. Some of them are passed parametes $rq and/or $rc. Both are lists. HTTP::LoadGen::Run exports constants to access the list elements. The structure of the request descriptor $rq is explained under URLList below. For the lack of a better place the $rc element is described here. RC_STATUS (0) the HTTP status code. If the request failed because the connection couldn't be established a code 599 is set here. "RC_STATUSLINE" describes the problem in more detail in that case. RC_STATUSLINE (1) the HTTP status message. If the server responds with the following first line for example: HTTP/1.1 501 Method Not Implemented "RC_STATUS" is 501 while "RC_STATUSLINE" is "Method Not Implemented". RC_HTTPVERSION (2) the server HTTP protocol version. Normally 1.1 or 1.0. RC_STARTTIME (3) when the request has been started, fractional number. RC_CONNTIME (4) when the connection has been established, fractional number. RC_FIRSTTIME (5) when the first line of output has been received, fractional number. RC_HEADERTIME (6) when the response HTTP header has been completely received, fractional number. RC_BODYTIME (7) when the response body has been completely received, fractional number. RC_HEADERS (8) a hash containing the response HTTP headers. The values of this hash are arrays since HTTP header fields can be given multiple times. Keys (header names) are converted to lower case. Example: { 'content-type' => ['text/html; charset=iso-8859-1'], 'connection' => ['close'], 'date' => ['Sun, 04 Jul 2010 18:21:12 GMT'], 'content-length' => ['217'], 'allow' => ['GET,HEAD,POST,OPTIONS,TRACE'], 'server' => ['Apache'], } RC_BODY (9) the response body RC_DNSCACHED (10) boolean: has the DNS cache lookup resulted in a hit (1) or miss (0)? RC_CONNCACHED (11) boolean: has the has a kept-alive connection been used? The %data hash So, what can be specified in %data? Note, all keys here are case sensitive. NWorker (optional) specifies the number of worker processes to be used. Default is 1. RampUpStart (optional) the number of threads to started up immediately (after the "ProcInit" phase is over). Default is 1 thread per worker process, that is "NWorker". RampUpMax (optional) the number of threads that have to be started up after the ramp-up phase is over. That means all processes together will start this number of threads. If a thread finishes before the ramp-up phase is over this maximum level of parallelism will never be reached. Default is the same as "RampUpStart". RampUpDuration (optional) the duration of the ramp-up phase in seconds (may be fraction). Default is 300 (5 minutes). ParentInit (optional) the "ParentInit" handler called as $data->{ParentInit}->(); One thing to do here is to create a scoreboard for interprocess communication, see HTTP::LoadGen::ScoreBoard or IPC::ScoreBoard. Example: ParentInit=>sub { # no parameters # create scoreboard # options() returns the config hash itself. The NWorker parameter # is known. SbSlotsz and SbExtra are new. This is to demonstrate # that the hook routines can access the configuration and evaluate # and even add custom parameters. HTTP::LoadGen::ScoreBoard::init_once @{HTTP::LoadGen::options()}{qw/NWorker SbSlotsz SbExtra/}; } ParentExit (optional) the "ParentExit" handler called as $data->{ParentExit}->(); If a scoreboard is used remember to disconnect. Example: ParentExit=>sub { # no parameters undef HTTP::LoadGen::ScoreBoard::scoreboard; } ProcInit (optional) the "ProcInit" handler called as $data->{ProcInit}->($procnr); $procnr is the 0 based number of the process. It ranges up to "NWorker - 1". If you plan to use the built-in random number generator this hook is a good place to reseed it. Another good thing to do here is to acquire a logger. HTTP::LoadGen::Logger may help here. If a scoreboard is used save $procnr as slot number. Example: ProcInit=>sub { my ($procnr)=@_; # set my slot number HTTP::LoadGen::ScoreBoard::slot=$procnr; # acquire a logger $logger=HTTP::LoadGen::Logger::get; } ProcExit (optional) the "ProcExit" handler called as $rc=$data->{ProcExit}->($procnr); The return value of this hook determines the exit code of the process. If omitted the exit code is 0. The thing to do here is perhaps to close the logger. Example: ProcExit=>sub { my ($procnr)=@_; $logger->(); # close the logger } ThreadInit (optional) the "ThreadInit" handler called as $userdata=$data->{ThreadInit}->(); The return value of this hook is saved as thread-specific user data. This hook is a good place to initialize a thread specific random number generator if you need repeatable results. Example: ThreadInit => sub { # no parameters # thread accounting with the scoreboard HTTP::LoadGen::ScoreBoard::thread_start; # set a thread specific RNG HTTP::LoadGen::rng=Math::Random::MT->new(@seed); return []; # initializes thread specific user data } ThreadExit (optional) the "ThreadExit" handler called as $data->{ThreadExit}->(); Remember to notify the scoreboard. Example: ThreadExit=>sub { # no parameters HTTP::LoadGen::ScoreBoard::thread_done; } ReqStart (optional) the "ReqStart" handler called as $data->{ReqStart}->($rq); $rq is an array specifying the current request. It is generated by the URL iterator. The "ReqStart" handler is allowed modify the array. If a scoreboard is used check in the request. One can also save some current state from the scoreboard to the thread-specific storage to log it later. Example: ReqStart=>sub { my ($rq)=@_; HTTP::LoadGen::ScoreBoard::req_start; @{HTTP::LoadGen::userdata()}=(HTTP::LoadGen::ScoreBoard::thread_count, HTTP::LoadGen::ScoreBoard::req_started, HTTP::LoadGen::ScoreBoard::req_success, HTTP::LoadGen::ScoreBoard::req_failed); } ReqDone (optional) the "ReqDone" handler called as $data->{ReqDone}->($rc, $rq); $rc is the request result. See "run_url" in HTTP::LoadGen::Run. The "ReqDone" handler may modify this array. But it's not recommended to do that. $rq is an array specifying the current request. Here you do request accounting and of course logging. Example: ReqDone=>sub { my ($rc, $rq)=@_; HTTP::LoadGen::ScoreBoard::req_done scalar($rc->[RC_STATUS]=~/^[23]/), $rc->[RC_HEADERS], $rc->[RC_BODY]; $logger->(HTTP::LoadGen::threadnr, @{$rc}[RC_DNSCACHED, RC_CONNCACHED], @{HTTP::LoadGen::userdata()}, HTTP::LoadGen::ScoreBoard::req_success, HTTP::LoadGen::ScoreBoard::req_failed, HTTP::LoadGen::ScoreBoard::header_count, HTTP::LoadGen::ScoreBoard::header_bytes, HTTP::LoadGen::ScoreBoard::body_bytes, $rc->[RC_STARTTIME], $rc->[RC_CONNTIME]-$rc->[RC_STARTTIME], $rc->[RC_FIRSTTIME]-$rc->[RC_STARTTIME], $rc->[RC_HEADERTIME]-$rc->[RC_STARTTIME], $rc->[RC_BODYTIME]-$rc->[RC_STARTTIME], $rc->[RC_STATUS], $rc->[RC_STATUSLINE], length($rc->[RC_BODY]), sprintf('%s(%s://%s:%s%s)', @{$rq}[RQ_METHOD, RQ_SCHEME, RQ_HOST, RQ_PORT, RQ_URI])); } times (optional) the number of times the URL iterator is charged. That many times the URL list is fetched. If omitted or "<=0" the test runs forever. dnscache (optional) "loadgen" caches DNS query results. One can prevent DNS queries completely in 2 ways. One of them is to provide a hash here that maps names to IP addresses. The other is to have the URL iterator generate IP addresses instead of host names and optionally "Host" request header fields. Another use of this item is to cheat host name resolution. One can for example test a newly installed or development server while the real server continues to work unaffected. Example: dnscache=>{ 'foertsch.name'=>'127.0.0.1', }, InitURLs (either InitURLs or URLList or both must be present) "InitURLs" initializes the URL iterator. It may be a string describing one of the predefined iterators or a "CODE" reference. In the latter case it is called without parameters as $it=$data->{InitURLs}->(); It is expected to return a function that when called as $new_rq=$it->($rc, $rq); returns the next request item or "undef" when it runs out of items. The parameters $rc and $rq describe the previous request ($rq) and its result ($rc). For a description of the $rq and $new_rq format see URLList below. Example: InitURLs=>sub { my $url=[qw!GET http foertsch.name 80 /-redir!, { keepalive=>KEEPALIVE, headers=>[ 'X-auth'=>1, # necessary to trigger 401 for that URL ], # it also shows a custom request header }]; return sub { my ($rc, $rq)=@_; if( $rc->[RC_STATUS]==401 ) { # redo with Authorization header push @{$rq->[RQ_PARAM]->{headers}}, Authorization=>'Basic YmxhOmJsdWI='; return $rq; } my $new_rq=$url; undef $url; # next time return undef (out-of-requests) return $new_rq; }; } The iterator generator initializes the variable $url and then returns a closure. Hence, $url is a static variable with respect to the returned iterator. The iterator itself checks the HTTP code of the previous request. In case of a 401 (Authorization Required) it adds an "Authorization" header to the request header list and retries the operation. If the previous operation has ended with an other HTTP code it copies $url to an auxiliary variable, undefines it and returns the auxiliary variable. Thus, only the first time the iterator is called it returns $url. After that it is always "undef" which signals *Out-of-Requests*. If "InitURLs" is a string it is the name of a predefined iterator generator. Example: InitURLs=>'follow' There are currently 4 such generators. All of them expect an "URLList" (see below) to be provided. default simply walks the "URLList" from start to end. This one is also used if "InitURLs" is omitted. random_start similar to "default" but starts at a random offset in "URLList". At the end of the list it continues at the beginning until all "URLList" elements are done once. follow similar to "default" but if a request results in a "3xx" HTTP code and a "Location" header is provided by the server it tries to follow it recursively. If the request starting a series of redirections contains a "postdelay" statement (see below) the delay is postponed until after the last request of the series. Subsequent requests are issues without delay. Subsequent requests inherit the "User-Agent" and "Referer" HTTP headers of the originating request, see follow_3XX() below. This iterator is a bit special in that can turn other iterators into following ones. Normally an iterator generator is called without parameters. This one can take one parameter that in turn may be an iterator. It returns then a following iterator based on the passed one. Infact, the built-in "random_start_follow" iterator is implemented for example as register_iterator random_start_follow=>sub { @_=get_iterator('random_start')->(); goto &{get_iterator 'follow'}; }; To turn your own iterator into a following you could write: InitURLs=>sub { return get_iterator->('follow')->($my_own_iterator); } where $my_own_iterator is an iterator function. random_start_follow a combination of the 2 above. You can register your own named iterators by calling register_iterator below. URLList (either InitURLs or URLList or both must be present) See also InitURLs above. An "URLList" is an array of arrays. Each of these sub-arrays describes one request. If consists of 6 elements: [$method, $scheme, $host, $port, $uri, $param] $method is the HTTP request method, e.g. "GET", "POST", ... $scheme is either "http" or "https". $host is the hostname or IP address of the server, e.g. "foertsch.name" or 109.73.51.50. $port is the server port to connect. Usually port 80 is used for "http" and port 443 for "https". $uri is the request URI normally starting with a slash ("/"), e.g. "/impressum.html". $param is a hash with further options. To access the elements of a request description HTTP::LoadGen::Run exports a few constants. They may be used to increase readability. RQ_METHOD == 0 RQ_SCHEME == 1 RQ_HOST == 2 RQ_PORT == 3 RQ_URI == 4 RQ_PARAM == 5 Example: URLList=>[ [qw!GET http 109.73.51.50 80 /-redir!, { keepalive=>KEEPALIVE, headers=>[ Authorization=>'Basic YmxhOmJsdWI=', Host=>'foertsch.name', ], }], [qw!HUGO https www.kabatinte.net 443 /!, { keepalive=>KEEPALIVE, predelay=>0.5, prejitter=>1, postdelay=>3, postjitter=>1.5, body=>'blablub', }] ] This "URLList" contains 2 requests, one for a server with the IP address 109.73.51.50 and one for the host "www.kabatinte.net". The first one will send the following HTTP request to the server (IP 109.73.51.50, port 80): GET /-redir HTTP/1.1 Authorization: Basic YmxhOmJsdWI= Host: foertsch.name If you need more header fields, "User-Agent" for example, add them to the "headers" array of the options hash. The second request is converted into the following HTTP message sent over SSL to "84.38.75.176:443" assuming that "www.kabatinte.net" resolves to 84.38.75.176: HUGO / HTTP/1.1 Host: www.kabatinte.net Content-Length: 7 blablub Although no "Host" header is specified in the request element one is sent. If the request element does not contain a "Host" header one is added automatically based on $host and $port. You may also notice the "Content-Length" header. It is sent because a request body is specified (the "body" item in $param). So, what can be specified in the $param part? keepalive HTTP::LoadGen::Run exports 3 constants to be used as values. "KEEPALIVE_USE" permits to use a previously kept alive connection. "KEEPALIVE_STORE" allows to keep the connection alive after the request. "KEEPALIVE" combines both of the above. If you hate readability you can also use the numerical values: KEEPALIVE_USE==1 KEEPALIVE_STORE==2 KEEPALIVE==3 predelay and prejitter These statements define a period to wait before sending the request. The wait is done after the request description has been pulled off the iterator but before the "ReqStart" handler is run. Both numbers can be fractions. Read them as predelay ± prejitter The actual waiting time is calculated as interval = predelay - prejitter + rand( 2 * prejitter ) If "prejitter >= predelay" interval can become negative. In this cases you won't jump back in time but simply not wait. To achieve repeatable results a thread-specific random number generator must be used. See the "rng" function below. postdelay and postjitter The same as "predelay" but waiting occurs after the request is done or more precisely after the "ReqDone" handler returns. headers an array (not a hash!) of header fields to be appended to the HTTP request. body a request body conn_timeout here you can specify the return value of the prepare-callback function passed to "AnyEvent::Socket::tcp_connect" when establishing a connection. See AnyEvent::Socket for more information. timeout the "timeout" parameter used when a connection is converted into a AnyEvent::Handle object. See AnyEvent::Handle for more information. tls_ctx the "tls_ctx" parameter used when a connection is converted into a AnyEvent::Handle object. See AnyEvent::Handle for more information. By now AnyEvent::Handle supports SSL features like client certificates and server certificate verification. However, some things are still missing like SSL session caching. How about server initiates renegotiations I am not sure. Note, "conn_timeout", "timeout" and "tls_ctx" are not very well tested by now. Useful functions to be used in hooks HTTP::LoadGen::threadnr returns the number of the thread currently running. HTTP::LoadGen::userdata returns the thread-specific user data. Normally this is assigned to by returning something useful from a "ThreadInit" handler. But it's a lvalue-function. Hence the following will work too: # assign new thread-specific data HTTP::LoadGen::userdata={something=>'useful'}; HTTP::LoadGen::options returns the copy of the configuration hash used by "loadgen()". HTTP::LoadGen::rng returns and sets the thread-specific random number generator. It sets the RNG used by "HTTP::LoadGen::rnd". HTTP::LoadGen::rnd $upper_limit use the thread-specific random number generator or if none set the built-in one. Returns a pseudo-random number. HTTP::LoadGen::delay $prefix, $param this function implements the "predelay" and "postdelay" operations. $prefix is a prefix, e.g. "pre" or "post". $param is a "RQ_PARAM" hash of a request descriptor containing keys "$prefix.'delay'" and "$prefix.'jitter'". HTTP::LoadGen::done if a thread needs a preliminary exit call HTTP::LoadGen::done=1 in a "ReqStart" or "ReqDone" handler. The current request will be performed except for "postdelay". Then the thread finishes. This can be used to stop the run when a certain load level has been reached. $new_rq=HTTP::LoadGen::follow_3XX $rc, $rq This function implements the following part of the built-in "follow" iterator. It is called with the result and the request descriptor of the previous request and returns a new request descriptor if the result is a HTTP redirect. Otherwise an empty list is returned. The new request preserves the "User-Agent" and "Referer" request header fields. Other auxiliary functions HTTP::LoadGen::register_iterator $name, $code_ref registers a known iterator. This can be used by other modules, e.g. package My::Iterator; use HTTP::LoadGen ':all'; BEGIN { register_iterator 'my_iterator'=>sub { ... }; } $code_ref=HTTP::LoadGen::get_iterator $name returns an iterator by name. $handle=HTTP::LoadGen::create_proc $nproc, $init_hnd, $hnd, $exit_hnd create $nproc child processes and have them finish the "ProcInit" phase $init_hnd. $init_hnd and $hnd are passed just one parameter, the 0-based process number. $exit_hnd get that plus the scalar return value of $hnd. The return value of $exit_hnd determines the exit code of the child process. $init_hnd and $exit_hnd may be "undef". $hnd may not. Returns an opaque handle that can be passed to "start_proc". $status=HTTP::LoadGen::start_proc $handle When "create_proc" returns all children have finished their $init_hnd and wait for a signal to continue with $hnd. "start_proc" sends that signal and waits for all children to finish. It returns a hash that maps operating system process IDs to their exit code, killing signal and a coredump flag. Example: { '7273' => [7, 0, 0], # PID 7273 exits normally with code 7 '7275' => [0, 11, 1], # PID 7275 has been killed by signal 11 # + core has been dumped '7274' => [8, 0, 0], # PID 7274 exits normally with code 8 } $semaphore=HTTP::LoadGen::ramp_up $procnr, $nproc, $start, $max, $duration, $handler implements the ramp-up phase. returns a semaphore that can be used to wait for the created threads to finish. It waits only for the threads running in the current process: $semaphore->down; # wait for my threads to finish "ramp_up" may finish almost immediately but may also take some time while the load generation is already running. It depends on the $duration parameter. Don't expect it to return before the load generation starts. Internal functions HTTP::LoadGen::conncache HTTP::LoadGen::tlscache DEBUGGING Sometimes its useful to see what requests are made. If the environment variable "HTTP__LoadGen__Run__dbg" is set when HTTP::LoadGen::Run is compiled a source filter is used to compile in debugging output to STDERR. EXPORT The following Exporter tags are defined: common exports "loadgen", "threadnr", "done", "userdata", "options", "rng", "rnd" and "delay" const exports all symbols that HTTP::LoadGen::Run exports by default. all all of the above. Additionally it pulls in HTTP::LoadGen::ScoreBoard and exports all that is exported by it. Also, HTTP::LoadGen::ScoreBoard is loaded. Then function named "get_logger" is created as an alias for "HTTP::LoadGen::Logger::get" and exported. EXAMPLE CONFIGURATION #!/usr/bin/loadgen # -*-perl-*- use strict; use Math::Random::MT; use Coro; use Coro::Timer (); no warnings 'ambiguous'; # possible hook parameters: # $procnr -- the current process number 0 .. NWorker-1 # $el -- an URL element to fetch (ARRAY) # use RQ_* constants from HTTP::LoadGen::Run to access # $rc -- an result element (ARRAY) # use RC_* constants from HTTP::LoadGen::Run to access my $logger; +{ NWorker=>3, # use 3 processes RampUpStart=>2, # start 2 threads immediately RampUpMax=>13, # then add 11 threads over 5 seconds RampUpDuration=>5, # that makes 2.2 new threads per second ParentInit=>sub { # no parameters # create scoreboard sbinit undef, options->{NWorker}; }, ParentExit=>sub { # no parameters undef scoreboard; }, ProcInit=>sub { my ($procnr)=@_; # set my slot number slot=$procnr; # acquire a logger my $fmt='%-2d %d %d %2d %2d %3d %3d %.3f %.3f %.3f %.3f %.3f %s %d '. "%s(%s://%s:%s%s) %s\n"; $logger=get_logger undef, sub {sprintf $fmt, @_}; }, ProcExit=>sub { my ($procnr)=@_; $logger->(); # close the logger }, ThreadInit=>sub { # no parameters # thread accounting thread_start; # set a thread specific RNG rng=Math::Random::MT->new(threadnr); return []; # initializes thread specific user data }, ThreadExit=>sub { # no parameters thread_done; }, ReqStart=>sub { my ($el)=@_; # request accounting req_start; # started - succeeded - failed = currently pending number of requests @{userdata()}=(thread_count, req_started-req_success-req_failed); }, ReqDone=>sub { my ($rc, $el)=@_; # request accounting: HTTP status 2xx and 3xx are successful # other requests are counted as failures. req_done +($rc->[RC_STATUS]=~/^[23]/), $rc->[RC_HEADERS], $rc->[RC_BODY]; $logger->(threadnr, @{$rc}[RC_DNSCACHED, RC_CONNCACHED], @{userdata()}, req_success, req_failed, $rc->[RC_STARTTIME], $rc->[RC_CONNTIME]-$rc->[RC_STARTTIME], $rc->[RC_FIRSTTIME]-$rc->[RC_STARTTIME], $rc->[RC_HEADERTIME]-$rc->[RC_STARTTIME], $rc->[RC_BODYTIME]-$rc->[RC_STARTTIME], $rc->[RC_STATUS], length($rc->[RC_BODY]), @{$el}[RQ_METHOD, RQ_SCHEME, RQ_HOST, RQ_PORT, RQ_URI], $rc->[RC_STATUSLINE]); }, dnscache=>{ localhost=>'127.0.0.1', 'kabatinte.net'=>'84.38.75.176', 'www.kabatinte.net'=>'84.38.75.176', 'foertsch.name'=>'109.73.51.50', }, times=>3, # run the URL list 3 times InitURLs=>'random_start', URLList=>do { my $o={ keepalive=>KEEPALIVE, qw!predelay 0.05 prejitter 0.1 postdelay 0.5 postjitter 1!, }; [[qw!GET http foertsch.name 80 /-redir!, $o], [qw!HUGO https www.kabatinte.net 443 /!, $o] ]; }, } SEE ALSO * HTTP::LoadGen::Run * HTTP::LoadGen::ScoreBoard * HTTP::LoadGen::Logger * loadgen AUTHOR Torsten Förtsch, COPYRIGHT AND LICENSE Copyright (C) 2010 by Torsten Förtsch This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.10.0 or, at your option, any later version of Perl 5 you may have available.