Speedup Minitest + RSpec + Turnip + Cucumber + Spinach by running parallel on multiple CPU cores.
ParallelTests splits tests into balanced groups (by number of lines or runtime) and runs each group in a process with its own database.
RailsCasts episode #413 Fast Tests
Gemfile:
gem 'parallel_tests', group: [:development, :test]ParallelTests uses 1 database per test-process.
| Process number | 1 | 2 | 3 |
| ENV['TEST_ENV_NUMBER'] | '' | '2' | '3' |
test:
database: yourproject_test<%= ENV['TEST_ENV_NUMBER'] %>rake parallel:create
rake parallel:create:<database>
rake parallel:create:secondary
rake parallel:prepare
rake parallel:migrate
rake parallel:migrate:<database>
rake parallel:setup
rake parallel:drop
rake parallel:drop:<database>
rake parallel:test # Minitest
rake parallel:spec # RSpec
rake parallel:features # Cucumber
rake parallel:features-spinach # Spinach
rake "parallel:test[1]" --> force 1 CPU --> 86 seconds
rake parallel:test --> got 2 CPUs? --> 47 seconds
rake parallel:test --> got 4 CPUs? --> 26 seconds
...
Test by pattern with Regex (e.g. use one integration server per subfolder / see if you broke any 'user'-related tests)
rake "parallel:test[^test/unit]" # every test file in test/unit folder
rake "parallel:test[user]" # run users_controller + user_helper + user tests
rake "parallel:test['user|product']" # run user and product related tests
rake "parallel:spec['spec\/(?!features)']" # run RSpec tests except the tests in spec/features
2 processes for 210 specs, ~ 105 specs per process
... test output ...
843 examples, 0 failures, 1 pending
Took 29.925333 seconds
RAILS_ENV=test parallel_test -e "rake my:custom:task"
# or
rake "parallel:rake[my:custom:task]"
# limited parallelism
rake "parallel:rake[my:custom:task,2]"require "parallel_tests"
# preparation:
# affected by race-condition: first process may boot slower than the second
# the Process.ppid will be the pod of the process that started the parallel tests
# when not using TEST_ENV_NUMBER we use a unique file per process because ppid would be the users shell
done = "/tmp/parallel-setup-done-#{ENV['TEST_ENV_NUMBER'] ? Process.ppid : Process.pid}"
if ParallelTests.first_process?
do_something
File.write done, "true"
else
sleep 0.1 until File.exist?(done)
end
# cleanup:
# could also use last_process? but that is just the last process to start, not the last to finish
at_exit do
if ParallelTests.first_process?
File.unlink done
ParallelTests.wait_for_other_processes_to_finish
undo_something
end
endTest groups will often run for different times, making the full test run as slow as the slowest group.
Step 1: Use these loggers (see below) to record test runtime
Step 2: The next test run will use the recorded test runtimes (use --runtime-log <file> if you picked a location different from below)
Step 3: Automate upload/download of test runtime from your CI system example (chunks need to be combined, an alternative is amend)
Rspec: Add to your .rspec_parallel (or .rspec), but can also be used via --test-options='--format x':
--format progress
--format ParallelTests::RSpec::RuntimeLogger --out tmp/parallel_runtime_rspec.log
To use a custom logfile location (default: tmp/parallel_runtime_rspec.log), use the CLI: parallel_test spec -t rspec --runtime-log my.log
Add to your test_helper.rb:
require 'parallel_tests/test/runtime_logger' if ENV['RECORD_RUNTIME']results will be logged to tmp/parallel_runtime_test.log when RECORD_RUNTIME is set,
so it is not always required or overwritten.
Log the test output without the different processes overwriting each other.
Add the following to your .rspec_parallel (or .rspec), but can also be used via --test-options='--format x':
--format progress
--format ParallelTests::RSpec::SummaryLogger --out tmp/spec_summary.log
Produce pasteable command-line snippets for each failed example. For example:
rspec /path/to/my_spec.rb:123 # should do somethingAdd the following to your .rspec_parallel (or .rspec), but can also be used via --test-options='--format x':
--format progress
--format ParallelTests::RSpec::FailuresLogger --out tmp/failing_specs.log
(Not needed to retry failures, for that pass --only-failures to rspec)
Prints a single line for starting and finishing each example, to see what is currently running in each process.
# PID, parallel process number, spec status, example description
[14403] [2] [STARTED] Foo foo
[14402] [1] [STARTED] Bar bar
[14402] [1] [PASSED] Bar bar
Add the following to your .rspec_parallel (or .rspec), but can also be used via --test-options='--format x':
--format ParallelTests::RSpec::VerboseLogger
Log failed cucumber scenarios to the specified file. The filename can be passed to cucumber, prefixed with '@' to rerun failures.
Usage:
cucumber --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log
Or add the formatter to the parallel: profile of your cucumber.yml:
parallel: --format progress --format ParallelTests::Cucumber::FailuresLogger --out tmp/cucumber_failures.log
but can also be used via --test-options='--format x':
Note if your cucumber.yml default profile uses <%= std_opts %> you may need to insert this as follows parallel: <%= std_opts %> --format progress...
To rerun failures:
cucumber @tmp/cucumber_failures.log
gem install parallel_tests
# go to your project dir
parallel_test
parallel_rspec
parallel_cucumber
parallel_spinach
-
use
ENV['TEST_ENV_NUMBER']inside your tests to select separate db/memcache/etc. (docker compose: expose it) -
Only run a subset of files / folders:
parallel_test test/bar test/baz/foo_text.rb -
Pass test-options and files via
--:parallel_rspec -- -t acceptance -f progress -- spec/foo_spec.rb spec/acceptance -
Pass in test options, by using the -o flag (wrap everything in quotes):
parallel_cucumber -n 2 -o '-p foo_profile --tags @only_this_tag or @only_that_tag --format summary'
Options are:
-n PROCESSES How many processes to use, default: available CPUs
-p, --pattern PATTERN run tests matching this regex pattern
--exclude-pattern PATTERN exclude tests matching this regex pattern
--group-by TYPE group tests by:
found - order of finding files
steps - number of cucumber/spinach steps
scenarios - individual cucumber scenarios
filesize - by size of the file
runtime - info from runtime log
default - runtime when runtime log is filled otherwise filesize
-m, --multiply-processes COUNT use given number as a multiplier of processes to run
-s, --single PATTERN Run all matching files in the same process
-i, --isolate Do not run any other tests in the group used by --single(-s)
--isolate-n PROCESSES Use 'isolate' singles with number of processes, default: 1
--highest-exit-status Exit with the highest exit status provided by test run(s)
--failure-exit-code INT Specify the exit code to use when tests fail
--specify-groups SPECS Use 'specify-groups' if you want to specify multiple specs running in multiple
processes in a specific formation. Commas indicate specs in the same process,
pipes indicate specs in a new process. If SPECS is a '-' the value for this
option is read from STDIN instead. Cannot use with --single, --isolate, or
--isolate-n. Ex.
$ parallel_tests -n 3 . --specify-groups '1_spec.rb,2_spec.rb|3_spec.rb'
Process 1 will contain 1_spec.rb and 2_spec.rb
Process 2 will contain 3_spec.rb
Process 3 will contain all other specs
--only-group GROUP_INDEX[,GROUP_INDEX]
Only run the given group numbers.
Changes `--group-by` default to 'filesize'.
-e, --exec COMMAND execute COMMAND in parallel and with ENV['TEST_ENV_NUMBER']
--exec-args COMMAND execute COMMAND in parallel with test files as arguments, for example:
$ parallel_tests --exec-args echo
> echo spec/a_spec.rb spec/b_spec.rb
-o, --test-options 'OPTIONS' execute test commands with those options
-t, --type TYPE test(default) / rspec / cucumber / spinach
--suffix PATTERN override built in test file pattern (should match suffix):
'_spec.rb$' - matches rspec files
'_(test|spec).rb$' - matches test or spec files
--serialize-stdout Serialize stdout output, nothing will be written until everything is done
--prefix-output-with-test-env-number
Prefixes test env number to the output when not using --serialize-stdout
--combine-stderr Combine stderr into stdout, useful in conjunction with --serialize-stdout
--non-parallel execute same commands but do not in parallel, needs --exec
--no-symlinks Do not traverse symbolic links to find test files
--ignore-tags PATTERN When counting steps ignore scenarios with tags that match this pattern
--nice execute test commands with low priority.
--runtime-log PATH Location of previously recorded test runtimes
--allowed-missing COUNT Allowed percentage of missing runtimes (default = 50)
--allow-duplicates When detecting files to run, allow duplicates
--unknown-runtime SECONDS Use given number as unknown runtime (otherwise use average time)
--first-is-1 Use "1" as TEST_ENV_NUMBER to not reuse the default test environment
--fail-fast Stop all groups when one group fails (best used with --test-options '--fail-fast' if supported
--test-file-limit LIMIT Limit to this number of files per test run by batching
(for windows set to ~100 to stay below 8192 max command limit, might have bugs from reusing test-env-number
and summarizing partial results)
--verbose Print debug output
--verbose-command Combines options --verbose-process-command and --verbose-rerun-command
--verbose-process-command Print the command that will be executed by each process before it begins
--verbose-rerun-command After a process fails, print the command executed by that process
--quiet Print only tests output
-v, --version Show Version
-h, --help Show this.
You can run any command in parallel with -e / --exec
parallel_test -n 3 -e 'ruby -e "puts %[hello from process #{ENV[:TEST_ENV_NUMBER.to_s].inspect}]"'
hello from process "2"
hello from process ""
hello from process "3"and pass arguments to a command with --exec-args
parallel_test -n 3 --exec-args echo
spec/a_spec.rb spec/b_spec.rb
spec/c_spec.rb spec/d_spec.rb
spec/e_spec.rband run multiple commands by using sh and --exec-args
parallel_test -n 3 --exec-args "sh -c \"echo 'hello world' && rspec \$@\" --"- Add a
.rspec_parallelto use different options, e.g. no --drb - Remove
--loadbyfrom.rspec - Instantly see failures (instead of just a red F) with rspec-instafail
- Use rspec-retry (not rspec-rerun) to rerun failed tests.
- JUnit formatter configuration
- Use parallel_split_test to run multiple scenarios in a single spec file, concurrently. (
parallel_testsworks at the file-level and intends to stay that way)
- Add a
parallel: fooprofile to yourconfig/cucumber.ymland it will be used to run parallel tests - ReportBuilder can help with combining parallel test results
- Supports Cucumber 2.0+ and is actively maintained
- Combines many JSON files into a single file
- Builds a HTML report from JSON with support for debug msgs & embedded Base64 images.
- [ZSH] use quotes to use rake arguments
rake "parallel:prepare[3]" - [Memcached] use different namespaces
e.g.config.cache_store = ..., namespace: "test_#{ENV['TEST_ENV_NUMBER']}" - Debug errors that only happen with multiple files using
--verboseand cleanser export PARALLEL_TEST_PROCESSORS=13to override default processor countexport PARALLEL_TEST_MULTIPLY_PROCESSES=.5to override default processor multiplier- Shell alias:
alias prspec='parallel_rspec -m 2 --' - [Spring] Add the spring-commands-parallel-tests gem to your
Gemfileto getparallel_testsworking with Spring. --first-is-1will make the first environment be1, so you can test while running your full suite.
export PARALLEL_TEST_FIRST_IS_1=truewill provide the same result- email_spec and/or action_mailer_cache_delivery
- zeus-parallel_tests
- Distributed Parallel Tests on CI systems) learn how
parallel_testscan run on distributed servers such as Travis and GitLab-CI. Also shows you how to use parallel_tests without addingTEST_ENV_NUMBER-backends - Capybara setup
- Sphinx setup
- Capistrano setup let your tests run on a big box instead of your laptop
- Rails vs
ArgumentError: secret_key_base: useconfig.secret_key_base = Random.hex(64), see rails issue
Contribute your own gotchas to the Wiki or even better open a PR :)
inspired by pivotal labs
- Charles Finkel
- Indrek Juhkam
- Jason Morrison
- jinzhu
- Joakim Kolsjö
- Kevin Scaldeferri
- Kpumuk
- Maksim Horbul
- Pivotal Labs
- Rohan Deshpande
- Tchandy
- Terence Lee
- Will Bryant
- Fred Wu
- xxx
- Levent Ali
- Michael Kintzer
- nathansobo
- Joe Yates
- asmega
- Doug Barth
- Geoffrey Hichborn
- Trae Robrock
- Lawrence Wang
- Sean Walbran
- Lawrence Wang
- Potapov Sergey
- Łukasz Tackowiak
- Pedro Carriço
- Pablo Manrubia Díez
- Slawomir Smiechura
- Georg Friedrich
- R. Tyler Croy
- Ulrich Berkmüller
- Grzegorz Derebecki
- Florian Motlik
- Artem Kuzko
- Zeke Fast
- Joseph Shraibman
- David Davis
- Ari Pollak
- Aaron Jensen
- Artur Roszczyk
- Caleb Tomlinson
- Jawwad Ahmad
- Iain Beeston
- Alejandro Pulver
- Felix Clack
- Izaak Alpert
- Micah Geisel
- Exoth
- sidfarkus
- Colin Harris
- Wataru MIYAGUNI
- Brandon Turner
- Matt Hodgson
- bicarbon8
- seichner
- Matt Southerden
- Stanislaw Wozniak
- Dmitry Polushkin
- Samer Masry
- Volodymyr Mykhailyk
- Mike Mueller
- Aaron Jensen
- Ed Slocomb
- Cezary Baginski
- Marius Ioana
- Lukas Oberhuber
- Ryan Zhang
- Rhett Sutphin
- Doc Ritezel
- Alexandre Wilhelm
- Jerry
- Aleksei Gusev
- Scott Olsen
- Andrei Botalov
- Zachary Attas
- David Rodríguez
- Justin Doody
- Sandeep Singh
- Calaway
- alboyadjian
- Nathan Broadbent
- Vikram B Kumar
- Joshua Pinter
- Zach Dennis
- Jon Dufresne
- Eric Kessler
- Adis Osmonov
- Josh Westbrook
- Jay Dorsey
- hatsu
- Mark Huk
- Johannes Vetter
Michael Grosser
michael@grosser.it
License: MIT