batchtools_slurm() now uses
makeClusterFunctionsSlurm2().Add support for controlling the ‘Rscript’ call in the built-in
job script templates. This can be done via fields rscript
and rscript_args of the resources
argument.
Add support for setting environment variables in the built-in job
script templates. This can be done via field envs of the
resources argument.
The built-in job script templates assert that the
Rscript launcher is found, and if not, they give an
informative error message suggesting to declare environment modules, via
the resources argument, that should be loaded by the job
script.
Add makeClusterFunctionsSlurm2(), which patches
batchtools::makeClusterFunctionsSlurm(). Firstly, it
patches the listJobsQueued() cluster function such that it
falls back to querying Slurm’s account database (sacct), if
the future was not found in the Slurm job queue
(squeue), which might be the case when Slurm provisions a
job that was just submitted to the scheduler. Secondly, it patched the
submitJob() cluster function such that the system call to
sbatch captures stderr separately from stdout, which
prevents auxiliary INFO messages from sbatch to corrupt the
output to be parsed.
batchtools_slurm() to
run R within a Linux container.batchtools_slurm() would produce “Future of class
BatchtoolsSlurmFuture expired, which indicates that it crashed or was
killed” errors on some Slurm clusters. We believe this happened because
a recently submitted future job would not immediately show up on the job
queue, which caused future.batchtools to incorrectly
conclude that the job had already finished, but without producing any
results. batchtools_slurm() now uses the new
makeClusterFunctionsSlurm2(), which does a better job
inferring whether a job is queued or not.
Most types of batchtools future can now be interrupted, including
batchtools_multicore and all job-scheduler backends, e.g.
batchtools_sge and batchtools_slurm.
Canceling batchtools futures will now interrupt them by default, if the backend supports it.
Calling plan() on batchtools_sge and
batchtools_slurm reports on the version of the
scheduler.
Printing a batchtools future will show logged output.
Built-in job-script templates:
Tidy up built-in template job scripts for Bash, SGE, Slurm and Torque/PBS.
Add built-in template job scripts for LSF and OpenLava.
All built-in template job scripts support “as-is” resource
specifications via character vector
resources[["asis"]].
All built-in template job scripts support “environment module”
resource specifications via character vector
resources[["modules"]]. When specified, corresponding
module load <name> entries are injected to the
generated job script.
All built-in template job scripts support “startup” and
“shutdown” code resource specifications via character vectors
resources[["startup"]] and
resources[["shutdown"]]. When specified, corresponding
lines are injected in the generated job script and the beginning and
end, respectively.
All built-in template job scripts support a “details” resource
specification via logical scalar resources[["details"]]. If
TRUE, scheduler job details are outputted to the log files at the
beginning and job summaries at the end.
All built-in template job scripts, which are written in Bash, error and exit early, by setting more strict Bash options. This should help with any troubleshooting, especially when getting started.
Batchtools future backends gained argument delete
for controlling if and when batchtools futures should be deleted from
the file system. Supported values are "on-success",
"never" and "always". The default value can be
set via R option future.batchtools.delete.
Add explicit batchtools arguments
scheduler.latency and fs.latency to batchtools
backends, where applicable.
Add timeout <runtime> ... to the default
batchtools_bash template script to illustrate how to
control this runtime via the backend resources
argument.
value() on a terminated batchtools job could
take a very long time before it returned with an error. This was because
it read logged output to be part of the error message. When there was no
output file, batchtools would wait for the file up to
fs.latency seconds (default 65 seconds) to give job
schedulers and any global file system time to write output to file. Now
future.batchtools will no longer wait for such files
and only read their content if they exists when checked.future.delete is deprecated. Please use new R
option future.batchtools.delete instead. For backward
compatible reasons, if R option future.delete sets option
future.batchtools.delete when the packages is loaded and
the latter is not already set. If future.delete is FALSE,
then future.batchtools.delete is set to
"never". If future.delete is TRUE, then
future.batchtools.delete is set to
"on-success".cancel()
would result in “Interruption of futures require a backend implementing
the FutureBackend API”. Until this package implements the new
FutureBackend API of future (>= 1.40.0), any calls to
cancel() will be silently ignored.plan(future.batchtools::batchtools_multicore) did not
support workers argument being a function.Improved performance of batchtools futures by avoiding re-checking the batchtools status if the batchtools job has already been observed to be resolved. Checking the batchtools status is fairly expense, especially since each status check queries a set of files on the file system.
Improved performance of batchtools futures by making the removal of the batchtools registry about 10-15 times faster.
run() for BatchtoolsFuture would update
the RNG state, if the future would attach packages.R_FUTURE_BATCHTOOLS_* environment variables are now
only read when the future.batchtools package is loaded,
where they set the corresponding future.batchtools* option.
This is in line with how all packages in the Futureverse works.
Add nbrOfFreeWorkers() for batchtools
futures.
It is now possible to tweak arguments used by an underlying
batchtools::makeClusterFunctionsNnn() function for some of
the batchtools_nnn backends, e.g.
plan(future.batchtools::batchtools_slurm, scheduler.latency = 60).
plan(future.batchtools::batchtools_multicore, workers = I(1))
overrides the fallback to batchtools_local and forces a
single batchtools_multicore worker.
print() for BatchtoolsFuture now reports on the
batchtools configuration file (an R script) and the the batchtools job
template file (a shell script) with info on location, file size, and
number of lines, if they exist.
run() for BatchtoolsFuture now produce an
informative BatchtoolsFutureError in case
batchtools::submitJobs() fails, for instance, due to
invalid job-scheduler resource specifications.
Add BatchtoolsFuture subclasses; abstract BatchtoolsUniprocessFuture, abstract BatchtoolsMultiprocessFuture, BatchtoolsInteractiveFuture, BatchtoolsLocalFuture, BatchtoolsMulticoreFuture, BatchtoolsLsfFuture, BatchtoolsOpenLavaFuture, BatchtoolsSGEFuture, BatchtoolsSlurmFuture, BatchtoolsTorqueFuture, and BatchtoolsCustomFuture.
Add batchtools_bash and BatchtoolsBashFuture to
illustrate how to create a basic batchtools_custom future
based on a batchtools template file
bash.tmpl part of the package.
Add batchtools_ssh and BatchtoolsSSHFuture for
creating batchtools futures based on batchtools SSH
workers created by
batchtools::makeClusterFunctionsSSH().
Add example template files for SGE and Slurm.
result() for HPC batchtools backends would use a
timeout of fs.latency seconds (as set for the cluster
functions) when trying to collect the logged output. However, since it
has already collected the results, the log file should be available
already and there would be no need to have to wait for the log file to
appear. Because of this, we temporarily set
fs.latency = 1.0 (second) timeout for trying to find the
log file. This makes a big difference in case the template used a
--output=<path> location other than
--output=<%= log.file %>. In such cases, the log file
would not be found, requiring a full timeout to complete.Using
plan(future.batchtools::batchtools_nnn, finalize = FALSE)
would give a warning on
Detected 1 unknown future arguments: 'finalize'.
Template files in
system.file(package = "future.batchtools", "templates")
were not found.
run(), resolved(), and
result() for BatchtoolsFuture would update the
RNG state.
Lazy batchtools futures only creates the internal batchtools registry when the future is launched.
Removed S3 generic functions await(),
finished(), and status(), which were functions
that were used for internal purposes.
future.delete and clarify option
future.cache.path in
help("future.batchtools.options").run() was called twice for a BatchtoolsFuture, it
would not produce a FutureError but only a regular non-classed
error.await(),
finished(), and status(), which were functions
that were used for internal purposes.It is now possible to configure the default number of workers on
the job queue of an HPC scheduler via either R option
future.batchtools.workers or environment variable
R_FUTURE_BATCHTOOLS_WORKERS.
It is now possible to configure the batchtools
registries that are used by batchtools futures via new argument
registry to plan(). This argument should be a
named list of parameters recognized by the batchtools
package,
e.g. plan(future.batchtools::batchtools_sge, registry = list(...)).
For notable example, see below news entries.
The default working directory for batchtools futures is the
current working directory of R when the batchtools future is
created. This corresponds to specifying
plan(future.batchtools::batchtools_nnn, registry = list(work.dir = NULL).
Sometimes it is useful to use a explicit working directory that is
guaranteed to be available on all workers on a shared file system,
e.g. plan(future.batchtools::batchtools_nnn, registry = list(work.dir = "~")).
It is possible to control if and how batchtools
should use file compression for exported globals and results by
specifying batchtools registry parameter
compress. For example, to turn off file compression, use
plan(future.batchtools::batchtools_nnn, registry = list(compress = FALSE)).
The default location of the .future folder can be
controlled by R option future.cache.path or environment
variable R_FUTURE_CACHE_PATH.
batchtools_custom() and BatchtoolsFuture gained
argument conf.file. Using
plan(future.batchtools::batchtools_custom) will now use any
batchtools configuration file (an R script) found on
the batchtools::findConfFile() search path.
help("future.batchtools.options") which
descriptions R options and environment variables used specifically by
the future.batchtools package.print() for BatchtoolsFuture would produce an error if
the underlying batchtools Registry was incomplete.Setting option future.delete to FALSE will now
prevent removal of the batchtools registry
folders.
When a batchtools job expires, for instance when
the scheduler terminates it because the job was running out of its
allocated resources, then a BatchtoolsFutureError is produced which by
default outputs the tail of the output logged by
batchtools. The default number of lines displayed from
the end is now increased from six to 48 - a number which now can be set
via option future.batchtools.expiration.tail.
Now a more informative error message is produced if a
batchtools *.tmpl template file was not
found.
Debug messages are now prepended with a timestamp.
Argument workers could not be a function.
Argument workers of type character was silently
accepted and effectively interpreted as
workers = length(workers).
example(future_custom).resolve() on a lazy batchtools future would stall and
never return.batchtools_* backends support the handling of the
standard output as implemented in future (>=
1.9.0).Error in readLog(id, reg = reg) : Log file for job with id 1 not available"
when using one of the batchtools backends. It occurred when the value
was queried. It was observed using batchtools_torque but
not when using batchtools_local. This bug was missed
because the 0.7.0 release was not tested on an TORQUE/PBS HPC scheduler
as it should have.workers of future strategies may now also be a
function, which is called without argument when the future strategy is
set up and used as-is. For instance,
plan(future.callr::callr, workers = halfCores) where
halfCores <- function() { max(1, round(availableCores() / 2)) }
will use half of the number of available cores. This is useful when
using nested future strategies with remote machines.If the built-in attempts of batchtools for
finding a default template file fails, then
system("templates", package = "future.batchtools") is
searched for template files as well. Currently, there exists a
torque.tmpl file.
A job’s name in the scheduler is now set as the future’s label (requires batchtools 0.9.4 or newer). If no label is specified, the default job name is controlled by batchtools.
The period between each poll of the scheduler to check whether a future (job) is finished or not now increases geometrically as a function of number of polls. This lowers the load on the scheduler for long running jobs.
The error message for expired batchtools futures now include the
last few lines of the logged output, which sometimes includes clues on
why the future expired. For instance, if a TORQUE/PBS job use more than
the allocated amount of memory it might be terminated by the scheduler
leaving the message
PBS: job killed: vmem 1234000 exceeded limit 1048576 in
the output.
print() for BatchtoolsFuture returns the object
invisibly.
Calling future_lapply() with functions containing
globals part of non-default packages would when using batchtools futures
give an error complaining that the global is missing. This was due to
updates in future (>= 1.4.0) that broke this
package.
loggedOutput() for BatchtoolsFuture would always
return NULL unless an error had occurred.
Added batchtools_custom() for specifying batchtools
futures using any type of batchtools cluster functions.
batchtools_template(pathname = NULL, type = <type>)
now relies on the batchtools package for locating the
<type> template file.
nbrOfWorkers() for batchtools futures now defaults
to +Inf unless the evaluator’s workers or
cluster.functions specify something else.
Renamed argument pathname to template
for batchtools_<tmpl>() functions.
plan(future.batchtools::batchtools_nnn), when
being created futures would produce an error on
all(is.finite(workers)) is not TRUE due to an outdated
sanity check.TESTS: Added test of future_lapply() for batchtools
backends.
TESTS: Added optional tests for batchtools_* HPC
schedulers listed in environment variable
R_FUTURE_TESTS_STRATEGIES.
batchtools_* backends is reported as +Inf. However, as
the number of workers is used by future_lapply() to decide
how many futures should be used to best partition the elements, this
means that future_lapply() will always use one future per
element. Because of this, it is now possible to specify
plan(future.batchtools::batchtools_nnn, workers = n) where
n is the target number of workers.