Running tests in parallel

Setup

To enable parallel testing, you must first be using the 3rd edition1. Then add the following line to the DESCRIPTION:

Config/testthat/parallel: true

By default, testthat will use getOption("Ncpus", 2) cores. To increase that value for your development machine we recommend setting TESTTHAT_CPUS in your .Renviron. The easiest way to do that is call usethis::edit_r_environ() and then add something like the following:

TESTTHAT_CPUS=4

Tests are run in alphabetical order by default, but you can often improve performance by starting the slowest tests first. Specify these tests by supplying a comma separated list of glob patterns2 to the Config/testthat/start-first field in your DESCRIPTION, e.g.:

Config/testthat/start-first: watcher, parallel*

Basic operation

Each worker begins by loading testthat and the package being tested. It then runs any setup files (so if you have existing setup files you’ll need to make sure they work when executed in parallel).

testthat runs test files in parallel. Once the worker pool is initialized, testthat then starts sending test files to workers, by default in alphabetical order: as soon as a subprocess has finished, it receives another file, until all files are done. This means that state is persisted across test files: options are not reset, loaded packages are not unloaded, the global environment is not cleared, etc. You are responsible for making sure each file leaves the world as it finds it.

Common problems

Performance

There is some overhead associated with running tests in parallel:

This overhead generally means that if you have many test files that take a short amount of time, you’re unlikely to see a huge benefit by using parallel tests. For example, testthat itself takes about 10s to run tests in serial, and 8s to run the tests in parallel.

Reporters

Default reporters

See default_reporter() for how testthat selects the default reporter for devtools::test() and testthat::test_local(). In short, testthat selects ProgressReporter for non-parallel and ParallelProgressReporter for parallel tests by default. (Other testthat test functions, like test_check(), test_file() , etc. select different reporters by default.)

Parallel support

Most reporters support parallel tests. If a reporter is passed to devtools::test(), testthat::test_dir(), etc. directly, and it does not support parallel tests, then testthat runs the test files sequentially.

Currently the following reporters don’t support parallel tests:

The other built-in reporters all support parallel tests, with some subtle differences:

Writing parallel reporters

To support parallel tests, a reporter must be able to function when the test files run in a subprocess. For example DebugReporter does not support parallel tests, because it requires direct interaction with the frames in the subprocess. When running in parallel, testthat does not provide location information (source references) for test successes.

To support parallel tests, a reporter must set self$capabilities$parallel_support to TRUE in its initialize() method:

...
initialize = function(...) {
  super$initialize(...)
  self$capabilities$parallel_support <- TRUE
  ...
}
...

When running in parallel, testthat runs the reporter in the main process, and relays information between the reporter and the test code transparently. (Currently the reporter does not even know that the tests are running in parallel.)

If a reporter does not support parallel updates (see below), then testthat internally caches all calls to the reporter methods from subprocesses, until a test file is complete. This is because these reporters are not prepared for running multiple test files concurrently. Once a test file is complete, testthat calls the reporter’s $start_file() method, relays all $start_test() , $end_test(), $add_result(), etc. calls in the order they came in from the subprocess, and calls $end_file() .

Parallel updates

The ParallelProgressReporter supports parallel updates. This means that once a message from a subprocess comes in, the reporter is updated immediately. For this to work, a reporter must be able to handle multiple test files concurrently.

A reporter declares parallel update support by setting self$capabilities$parallel_updates to TRUE:

...
initialize = function(...) {
  super$initialize(...)
  self$capabilities$parallel_support <- TRUE
  self$capabilities$parallel_updates <- TRUE
  ...
}
...

For these reporters, testthat does not cache the messages from the subprocesses. Instead, when a message comes in:

testthat also calls the new $update() method of the reporter regularly, even if it does not receive any messages from the subprocess. (Currently aims to do this every 100ms, but there are no guarantees.) The $update() method may implement a spinner to let the user know that the tests are running.


  1. See vignette("third-edition") for details.↩︎

  2. See ?utils::glob2rx for details↩︎